ANT-2026-Y9CTVKPP · jetty/jetty.project

denial-of-service high

Severity Claude high · Security research firm high · Maintainer -

Discovered by Claude Mythos Preview

REPORT

Anthropic's analysis, sealed at approval. Disclosure to the maintainer was performed by Ada Logics.

ANT-2026-Y9CTVKPP: WebSocket reserved opcode bypasses frame size limits

In Jetty's WebSocket core Parser, checkFrameSize() only enforces maxFrameSize for control or data opcodes, and the else-branch check is disabled when autoFragment is true (the default). Reserved opcodes 0x03-0x07 and 0x0B-0x0F therefore skip both the size check and the auto-fragment clamps in parsePayload(), and execution reaches bufferPool.acquire(payloadLength) at Parser.java:346 with an attacker-controlled 63-bit length truncated to int (up to Integer.MAX_VALUE). ArrayByteBufferPool falls through to a raw ByteBuffer.allocate of ~2 GiB, and the illegal opcode is only rejected afterward in newFrame()/FrameSequence. A single ~15-byte frame header sent after a WebSocket upgrade thus forces a ~2 GiB heap allocation per connection, allowing an unauthenticated remote attacker to drive the JVM to OutOfMemoryError and take down every hosted application.

Target

Project: jetty/jetty.project
Location: jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java:346
Discovery: static analysis — not yet dynamically reproduced

Technical Details

OpCode.isControlFrame() and OpCode.isDataFrame() both return false for reserved opcodes, so Parser.checkFrameSize() (lines 265-275) falls into an else branch whose throw is gated by !configuration.isAutoFragment() — with DEFAULT_AUTO_FRAGMENT=true no limit is applied. parsePayload()'s fragment clamps at lines 333/341 require isDataFrame==true and are also skipped, so aggregate = bufferPool.acquire(payloadLength, false) at line 346 runs with attacker-controlled payloadLength before the only opcode validation (OpCode.isKnown() in newFrame(), line 282) executes.

Reproduction

Complete a WebSocket handshake against the target Jetty endpoint.
Send a frame with FIN=1, opcode=0x03 (reserved), MASK=1, payload-len=127, 8-byte extended length = 0x000000007FFFFFF0, a 4-byte masking key, and at least 1 payload byte.
Parser.parsePayload() calls bufferPool.acquire(~2 GiB) before validating the opcode.
Repeat over a handful of concurrent connections to exhaust the JVM heap and trigger OutOfMemoryError.

[No reproducer or sanitizer output attached — request from security-cvd@anthropic.com if needed.]

Suggested Fix

Reject frames with reserved/unknown opcodes before any payload-length-dependent processing, and enforce the configured maximum frame size unconditionally for all opcode categories.

Acknowledgement

This vulnerability was discovered by Claude, Anthropic's AI assistant, and triaged by the Anthropic security team in collaboration with Anthropic Research. Please direct questions to security-cvd@anthropic.com and reference ANT-2026-Y9CTVKPP.

Reference: ANT-2026-Y9CTVKPP
Anthropic CVD Policy: https://www.anthropic.com/coordinated-vulnerability-disclosure

SECURITY RESEARCH FIRM ANALYSIS

Triage and disclosure were performed by Ada Logics. The writeup below is the document the firm sent to the maintainer.

Verdict: true positive
Severity: high

Jetty's WebSocket frame parser rejects an oversized frame and validates the frame opcode at two different points, and for a reserved opcode the ordering lets an attacker force a multi-gigabyte allocation before the opcode is ever checked. When Parser.parse() has read a frame header it calls checkFrameSize(opcode, payloadLength), but the non-control branch of that method only enforces maxFrameSize when auto-fragmentation is disabled, and auto-fragmentation is on by default. It then calls parsePayload(), whose auto-fragment clamps apply only to data frames (CONTINUATION/TEXT/BINARY). A reserved opcode such as 0x03 is neither a control frame nor a data frame, so it slips past both guards and reaches aggregate = bufferPool.acquire(payloadLength, false), allocating the full attacker-declared payload length up to Integer.MAX_VALUE in one go. The opcode is only validated later, inside newFrame() via OpCode.isKnown(), which is reached after the allocation has already been requested. A single WebSocket frame header of about fifteen bytes therefore forces a roughly 2 GiB heap allocation and causes OutOfMemoryError.

Root cause

Parser.parse() reads the frame header, decodes the declared payloadLength (the 8-byte extended length is masked down to a non-negative int, so it can be as large as Integer.MAX_VALUE), and calls the size checking logic.

https://github.com/jetty/jetty.project/blob/852c52def077bd7878807b2534c36cb618f7b03a/jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java#L221

checkFrameSize method rejects oversized control frames, but for everything else the only throw is gated on auto-fragmentation being off.

Auto-fragmentation is on by default.

Configuration.isAutoFragment() returns that default when the value is unset.

So with the shipped defaults the !isAutoFragment() condition is false and the maxFrameSize rejection is skipped for every non-control frame, including reserved opcodes.

The design intent is that the size limit is enforced later, by clamping through auto-fragmentation in parsePayload(). But that clamping is conditined on the frame being a data frame.

OpCode.isDataFrame() returns true only for CONTINUATION, TEXT and BINARY, the other opcode is considered reserved and not a data frae.

For a reserved opcode such as 0x03, isDataFrame is false, so both auto-fragment paths (the size clamp at line 333 and the insufficient-data clamp at line 341) are skipped, and execution falls straight through to the bufferPool.acquire(payloadLength, false) call at line 346 shown above, which requests a buffer of the full declared length.

The opcode is only validated afterwards, inside newFrame().

OpCode.isKnown() would throw ProtocolException for the reserved opcode 0x03 by returning false.

That check comes too late: the roughly 2 GiB acquire at earlier line has already run. The reserved opcode is the exact input that evades both the maxFrameSize rejection and the auto-fragment clamp, so it is the one case that reaches an unbounded allocation. A legitimate BINARY frame with the same declared length is safe precisely because isDataFrame is true and it is clamped to maxFrameSize

The parser runs on the connection read path with the live session configuration. WebSocketConnection constructs its Parser from the core session.

It drives the parser from onFillable().

That path reaches the parser.parse(...) call on the network buffer, all before the frame is delivered to any application handler.

Proof of Concept

The reproducer drives the real upstream Parser with a default Configuration (autoFragment=true, maxFrameSize=64KiB) and feeds it two single frames: a negative control with data opcode 0x02 (BINARY) and the attack frame with reserved opcode 0x03, both declaring a payload length of 0x7FFFFFF0 (about 2 GiB). It runs under -Xmx256m so that the unbounded allocation deterministically fails with OutOfMemoryError, while the clamped control survives, and it unwinds the thrown exception to print the Jetty allocation stack. This is a component-level harness against the shipped parser; the network path that reaches it (onFillable to parse) is established by source above rather than exercised over a socket.

Harness.java

FROM ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive

ARG TARGET_COMMIT=95efeb2bf041bc6b7a5da91b281499592417371b

RUN apt-get update && apt-get install -y --no-install-recommends \
        ca-certificates git openjdk-17-jdk-headless maven \
    && rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/jetty/jetty.project /src/repo
WORKDIR /src/repo
RUN git checkout ${TARGET_COMMIT}

# Build jetty-websocket-core-common
RUN mvn -q -B -P'!config' -pl jetty-core/jetty-websocket/jetty-websocket-core-common -am install \
        -DskipTests -Dmaven.test.skip=true \
        -Dcheckstyle.skip=true -Dspotbugs.skip=true -Denforcer.skip=true \
        -Dmaven.javadoc.skip=true -Dlicense.skip=true -Dsource.skip=true \
        -Dmaven.source.skip=true -Djapicmp.skip=true

# Assemble the runtime classpath of the module into /tmp/cp.txt.
RUN mvn -q -B -pl jetty-core/jetty-websocket/jetty-websocket-core-common \
        dependency:build-classpath -Dmdep.outputFile=/tmp/cp.txt -Denforcer.skip=true

# Compile the reproducer harness against the freshly built module + its deps.
ENV MODCLASSES=/src/repo/jetty-core/jetty-websocket/jetty-websocket-core-common/target/classes
COPY Harness.java /tmp/Harness.java
RUN javac -cp "${MODCLASSES}:$(cat /tmp/cp.txt)" -d /tmp /tmp/Harness.java

# Small heap so the unbounded ~2 GiB allocation deterministically OOMs (proving the
# allocation is unbounded), while the negative control (clamped to 64KiB) survives.
CMD ["/bin/sh","-c","java -Xmx256m -cp \"/tmp:${MODCLASSES}:$(cat /tmp/cp.txt)\" Harness 2>&1; echo EXIT=$?"]

docker build -t poc . && docker run --rm poc

Result

Heap max: 256 MiB

--- NEGATIVE CONTROL: data opcode 0x02 (BINARY) ---
[control] opcode=0x02 declared payloadLength=2147483632 (2047 MiB)
[control] sending 15-byte frame header, then parse()...
[control] parse() returned without large allocation (clamped/auto-fragmented)

--- ATTACK: reserved opcode 0x03 ---
[attack] opcode=0x03 declared payloadLength=2147483632 (2047 MiB)
[attack] sending 15-byte frame header, then parse()...
[attack] *** OutOfMemoryError *** triggered by a 15-byte frame header: Java heap space
[attack] VULNERABILITY CONFIRMED: bufferPool.acquire(2147483632) ran before opcode validation. Allocation site:
    at org.eclipse.jetty.util.BufferUtil.allocate(BufferUtil.java:127)
    at org.eclipse.jetty.util.BufferUtil.allocate(BufferUtil.java:158)
    at org.eclipse.jetty.io.ArrayByteBufferPool.acquire(ArrayByteBufferPool.java:241)
    at org.eclipse.jetty.websocket.core.internal.Parser.parsePayload(Parser.java:346)
    at org.eclipse.jetty.websocket.core.internal.Parser.parse(Parser.java:222)

The reserved-opcode frame reaches bufferPool.acquire(2147483632) at Parser.parsePayload and throws OutOfMemoryError under the small heap, while the identical data-opcode frame is clamped by auto-fragmentation and returns without a large allocation. In a live server the same roughly 2 GiB allocation is held per connection while the parser awaits the remainder of the frame, so a small number of concurrent frames exhausts the heap and denies service across the whole JVM.

Mitigation

Validate the opcode before allocating for the payload: call OpCode.isKnown() at the start of checkFrameSize() or before parsePayload() acquires a buffer, so an unknown opcode is refused early while only the header has been read.

Attribution

This vulnerability was discovered by Claude, Anthropic's AI assistant, and triaged manually with manual report writing by Ada Logics in collaboration with Anthropic Research.

UPSTREAM FIX

The change that resolved this finding.

diff --git a/jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java b/jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java
index 7626f40a596..184a53b26ee 100644
--- a/jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java
+++ b/jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java
@@ -107,6 +107,7 @@ public Frame.Parsed parse(ByteBuffer buffer) throws WebSocketException
                         // peek at byte
                         firstByte = buffer.get();
                         state = State.PAYLOAD_LEN;
+                        checkFirstByte(firstByte);
                         break;
                     }
 
@@ -257,6 +258,19 @@ else if (payloadLength == 0)
         return null;
     }
 
+    protected void checkFirstByte(byte firstByte)
+    {
+        // Validate OpCode
+        byte opcode = OpCode.getOpCode(firstByte);
+        if (!OpCode.isKnown(opcode))
+            throw new ProtocolException("Unknown opcode: " + opcode);
+
+        // Validate Control Frame
+        boolean fin = ((firstByte & 0x80) != 0);
+        if (OpCode.isControlFrame(opcode) && !fin)
+            throw new ProtocolException("Fragmented Control Frame [" + OpCode.name(opcode) + "]");
+    }
+
     protected void checkFrameSize(byte opcode, int payloadLength) throws MessageTooLargeException, ProtocolException
     {
         if (payloadLength < 0)
@@ -267,26 +281,20 @@ protected void checkFrameSize(byte opcode, int payloadLength) throws MessageTooL
             if (payloadLength > Frame.MAX_CONTROL_PAYLOAD)
                 throw new ProtocolException("Invalid control frame payload length, [" + payloadLength + "] cannot exceed [" + Frame.MAX_CONTROL_PAYLOAD + "]");
         }
-        else
+        else if (OpCode.isDataFrame(opcode))
         {
             long maxFrameSize = configuration.getMaxFrameSize();
             if (!configuration.isAutoFragment() && maxFrameSize > 0 && payloadLength > maxFrameSize)
                 throw new MessageTooLargeException("Cannot handle payload lengths larger than " + maxFrameSize);
         }
+        else
+        {
+            throw new ProtocolException("Unknown opcode: " + opcode);
+        }
     }
 
     protected Frame.Parsed newFrame(byte firstByte, byte[] mask, ByteBuffer payload, Runnable releaser)
     {
-        // Validate OpCode
-        byte opcode = OpCode.getOpCode(firstByte);
-        if (!OpCode.isKnown(opcode))
-            throw new ProtocolException("Unknown opcode: " + opcode);
-
-        // Validate Control Frame
-        boolean fin = ((firstByte & 0x80) != 0);
-        if (OpCode.isControlFrame(opcode) && !fin)
-            throw new ProtocolException("Fragmented Control Frame [" + OpCode.name(opcode) + "]");
-
         return new Frame.Parsed(firstByte, mask, payload, releaser);
     }
 
diff --git a/jetty-core/jetty-websocket/jetty-websocket-core-server/src/main/java/org/eclipse/jetty/websocket/core/server/internal/RFC6455Handshaker.java b/jetty-core/jetty-websocket/jetty-websocket-core-server/src/main/java/org/eclipse/jetty/websocket/core/server/internal/RFC6455Handshaker.java
index a9a2c19c0a4..1742c10ba81 100644
--- a/jetty-core/jetty-websocket/jetty-websocket-core-server/src/main/java/org/eclipse/jetty/websocket/core/server/internal/RFC6455Handshaker.java
+++ b/jetty-core/jetty-websocket/jetty-websocket-core-server/src/main/java/org/eclipse/jetty/websocket/core/server/internal/RFC6455Handshaker.java
@@ -77,10 +77,11 @@ protected boolean validateNegotiation(WebSocketNegotiation negotiation)
     @Override
     protected WebSocketConnection createWebSocketConnection(Request baseRequest, WebSocketCoreSession coreSession)
     {
+        WebSocketComponents components = coreSession.getWebSocketComponents();
         ConnectionMetaData connectionMetaData = baseRequest.getConnectionMetaData();
         Connector connector = connectionMetaData.getConnector();
         EndPoint endPoint = connectionMetaData.getConnection().getEndPoint();
-        return newWebSocketConnection(endPoint, connector.getExecutor(), connector.getScheduler(), connector.getByteBufferPool(), coreSession);
+        return newWebSocketConnection(endPoint, components.getExecutor(), connector.getScheduler(), components.getByteBufferPool(), coreSession);
     }
 
     @Override

https://github.com/jetty/jetty.project/commit/b5bdc24629cac46ed51c12de11afd7a068cb3b48

TIMELINE

Recorded dates, in order.

2026-04-02 Discovered or logged
2026-07-22 Sent to maintainer
2026-07-22 Maintainer acknowledged
2026-08-04 Patch released
2026-09-28 Publicly revealed

PROVENANCE

SHA-3-512 hash:

02d2070d13dafd3b1d7788dc77f0547d32da15bb8053ba64bb1f63a13403631f7b91237816ab5daa27b3b62ddab2fe85b44cfa74b17cc7805f0816d724029a71

Committed 2026-07-22 07:33 UTC

Revealed 2026-09-28 22:29 UTC

Verify (download preimage.json)

Show preimage JSON

{
  "ant_id": "ANT-2026-Y9CTVKPP",
  "bug_class": "Denial of Service / Resource Exhaustion",
  "claude_severity": "high",
  "commit_sha": null,
  "created_at": "2026-04-16T02:39:24+00:00",
  "description": "In Jetty's WebSocket core Parser, checkFrameSize() only enforces maxFrameSize for control or data opcodes, and the else-branch check is disabled when autoFragment is true (the default). Reserved opcodes 0x03-0x07 and 0x0B-0x0F therefore skip both the size check and the auto-fragment clamps in parsePayload(), and execution reaches bufferPool.acquire(payloadLength) at Parser.java:346 with an attacker-controlled 63-bit length truncated to int (up to Integer.MAX_VALUE). ArrayByteBufferPool falls through to a raw ByteBuffer.allocate of ~2 GiB, and the illegal opcode is only rejected afterward in newFrame()/FrameSequence. A single ~15-byte frame header sent after a WebSocket upgrade thus forces a ~2 GiB heap allocation per connection, allowing an unauthenticated remote attacker to drive the JVM to OutOfMemoryError and take down every hosted application.",
  "discovered_at": "2026-04-02T00:00:00+00:00",
  "location": "jetty-core/jetty-websocket/jetty-websocket-core-common/src/main/java/org/eclipse/jetty/websocket/core/internal/Parser.java:346",
  "poc_sha256": null,
  "preimage_version": 1,
  "project": "jetty/jetty.project",
  "reproduction": [
    "1. Complete a WebSocket handshake against the target Jetty endpoint.",
    "2. Send a frame with FIN=1, opcode=0x03 (reserved), MASK=1, payload-len=127, 8-byte extended length = 0x000000007FFFFFF0, a 4-byte masking key, and at least 1 payload byte.",
    "3. Parser.parsePayload() calls bufferPool.acquire(~2 GiB) before validating the opcode.",
    "4. Repeat over a handful of concurrent connections to exhaust the JVM heap and trigger OutOfMemoryError."
  ],
  "technical_details": "OpCode.isControlFrame() and OpCode.isDataFrame() both return false for reserved opcodes, so Parser.checkFrameSize() (lines 265-275) falls into an else branch whose throw is gated by !configuration.isAutoFragment() — with DEFAULT_AUTO_FRAGMENT=true no limit is applied. parsePayload()'s fragment clamps at lines 333/341 require isDataFrame==true and are also skipped, so aggregate = bufferPool.acquire(payloadLength, false) at line 346 runs with attacker-controlled payloadLength before the only opcode validation (OpCode.isKnown() in newFrame(), line 282) executes.",
  "title": "WebSocket reserved opcode bypasses frame size limits",
  "vendor_severity": "high"
}