[jira] [Updated] (CAMEL-25573) camel-netty-http - unmarshal().gzipDeflater() of a request body fails with "Input is not in the .gz format", and a byte-by-byte read of the body ends at the first 0xFF byte or throws at its end

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25573:
--------------------------------
    Fix Version/s: 4.24.0

> camel-netty-http - unmarshal().gzipDeflater() of a request body fails with "Input is not in the .gz format", and a byte-by-byte read of the body ends at the first 0xFF byte or throws at its end
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25573
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25573
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-netty-http
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> The body of a request received by a netty-http consumer is a {{NettyChannelBufferStreamCache}} ({{DefaultNettyHttpBinding.toCamelMessage}}, lines 119-123 of main 2a0a2b54f4ba; not with {{disableStreamCache=true}} or {{httpProxy=true}}). The class is a {{java.io.InputStream}}, and a route step that asks for the body as an {{InputStream}} gets this object. Its {{read()}} does not keep the {{InputStream}} contract. With the default configuration:
> # Loud: {{from("netty-http:...").unmarshal().gzipDeflater()}} does not work. Every request fails with HTTP 500 and {{java.io.IOException: Input is not in the .gz format.}} in the log. The same holds for any reader that reads the first bytes one by one (to sniff a header, for example): it gets wrong values for bytes from 0x80, or an {{IndexOutOfBoundsException}} at the end of the body.
> # Silent, and narrower: user code or a library that reads the body byte by byte ({{while ((b = in.read()) != -1)}}) gets only the bytes before the first 0xFF, without any error, and negative values for bytes from 0x80. A byte 0xFF is common in binary payloads and never occurs in UTF-8 text, so text bodies are not cut; for them the loop ends with the {{IndexOutOfBoundsException}} (loud).
> Not affected: everything that reads in bulk. The type converters of Camel to {{byte[]}} and {{String}} ({{StreamCacheConverter.convertToByteArray}} calls {{writeTo}}) and {{NettyHttpConverter.toByteBuf}} read with {{read(byte[], int, int)}}, which is why most routes do not see this. Silent loss is not the general case.
> The causes in the class:
> * {{read()}} (line 56) is {{return buffer.readByte();}}. {{ByteBuf.readByte()}} returns a Java {{byte}} (-128 to 127) and throws {{IndexOutOfBoundsException}} when nothing is left. {{InputStream.read()}} must return "an int in the range 0 to 255" and -1 at the end of the stream. So a byte 0xFF is returned as -1, which every reader takes for the end of the stream, bytes 0x80 to 0xFE are returned as negative values, and at the end of the body the method throws instead of returning -1.
> * {{markSupported()}} (line 51) returns true, but {{mark(int)}} is not overridden (it is the empty method of {{InputStream}}). The gzip reader of commons-compress uses a stream with {{markSupported()}} as it is, so it reads the magic number with {{read()}} and gets -117 for the byte 0x8b. ({{java.util.zip.GZIPInputStream}} fails with {{NettyChannelBufferStreamCache.read() returned value out of range -1..255: -117}}.)
> * {{read(byte[], int, int)}} (lines 65-78) takes {{buffer.capacity()}} for the end of the content; for a buffer whose capacity is larger than its writer index it throws {{IndexOutOfBoundsException}} at the end of the content. (The aggregated content of a request has no spare capacity, so this one is not reached by the consumer today.)
> h3. Reproduction
> New {{NettyHttpGzipRequestBodyTest}} (4 tests, real consumer and producer) and {{NettyChannelBufferStreamCacheTest}} (7 tests on the class). With the class of main 10 of the 11 fail:
> * {{gzipDataFormatUnmarshalsTheRequestBody}}, {{resetReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}}, {{writeToReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}}: {{NettyHttpOperationFailedException ... statusCode: 500}}, in the log of the consumer {{java.io.IOException: Input is not in the .gz format.}} (all three fail at the unmarshal)
> * {{processorReadsTheRequestBodyByteByByte}} (body {{10 20 FF 30 40}}, the route counts the bytes with {{read()}}): {{expected: <5> but was: <2>}}
> * {{readReturnsEveryByteAsUnsignedValue}}: {{expected: <128> but was: <-128>}}
> * {{readLoopCopiesBinaryContent}}: {{array lengths differ, expected: <5> but was: <2>}}
> * {{readLoopEndsAtTheEndOfText}}: {{IndexOutOfBounds readerIndex(5) + length(1) exceeds writerIndex(5)}}
> * {{markIsNotSupported}}: {{expected: <false> but was: <true>}}
> * {{readStopsAtTheLastWrittenByte}}: {{IndexOutOfBounds readerIndex(0) + length(16) exceeds writerIndex(3)}}
> * {{gzipInputStreamReadsItsHeader}}: {{IOException: ...NettyChannelBufferStreamCache.read() returned value out of range -1..255: -117}}
> {{resetAndWriteToReturnTheWholeBodyAfterAReaderThatMarks}} passes on main too (there {{mark}} does nothing either); it guards the fix against a {{mark(int)}} that moves the reset position, see below.
> h3. Proposed fix
> * {{read()}} returns {{buffer.readByte() & 0xFF}}, and -1 when the buffer has no readable byte.
> * {{markSupported()}} returns false. {{mark(int)}} stays the empty method and {{reset()}} is not changed: it goes back to the start of the body, as {{StreamCache.reset()}} needs. A reader that needs mark support wraps the stream (the gzip reader of commons-compress does that by itself with a {{BufferedInputStream}}).
> * {{read(byte[], int, int)}} uses {{buffer.readableBytes()}}.
> Why not implement {{mark(int)}}: {{reset()}}, {{writeTo()}} (used by the converters to {{byte[]}} and {{String}}) and {{NettyHttpConverter.toByteBuf}} all go back to the marked reader index of the buffer, which today is always the start because only the constructor marks. If {{mark(int)}} moved it, a reader that marks in the middle (the gzip reader marks before every buffer of 8192 bytes) would leave the stream cache returning only its tail: with that variant the tests got 3634 of 20028 bytes from {{reset()}} and {{writeTo()}} after the gzip unmarshal. A stream cache without mark support whose {{reset()}} goes back to the start exists already: {{FileInputStreamCache}} (line 59, no {{markSupported}} or {{mark}} override; {{reset()}} at line 89 drops the open stream, so the next read starts at the beginning of the file). The in-memory stream caches of camel-support differ: they are, or delegate to, a {{ByteArrayInputStream}}, which supports mark.
> Behaviour change: code that reads the body byte by byte now gets values from 0 to 255 and -1 at the end, and {{markSupported()}} is false. An upgrade note for 4.24 is included (the fix creates {{camel-4x-upgrade-guide-4_24.adoc}}). In the main sources of Camel the only caller of {{markSupported()}} on a body stream that can be this class is {{BlobClientWrapper}} of camel-azure-storage-blob (line 137; read, not run). camel-netty-http with the fix: 312 tests, 0 failures, 8 skipped (skipped before as well).
> Not run: a byte-by-byte reader of a text body in a route (the exception at the end is shown on the class only), a dead letter channel with {{useOriginalMessage}}.
> Related, and independent of this ticket: the gzip data format of camel-zip-deflater moving the mark of the in-memory stream caches of camel-support has its own ticket. This fix does not rely on that one.
> Found with a Lean 4 model of {{read()}} and of the copy loop of a reader: "the reader gets the whole body" fails for every body with a byte 0xFF (the reader ends normally with the bytes before it); a body without one ends with the exception in the tested case. For the fixed {{read()}} the copy loop is proved to return every body, and main and the fix are proved equal for bytes below 0x80. For {{reset()}} the model shows that no sequence of read, mark and reset calls of a reader changes where it goes back to (by construction: {{mark}} does nothing), and gives the 3634 of 20028 bytes for the variant with a marking {{mark(int)}}. Then confirmed with the consumer as above.
> Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same lines (read, not run).
> Duplicate check (2026-10-12): JIRA text "NettyChannelBufferStreamCache" (5: CAMEL-7884, CAMEL-8284, CAMEL-8408, CAMEL-13967, CAMEL-14166, none about {{read()}} or {{mark}}), "netty-http markSupported" (none), "not in the .gz format" (none), "netty-http gzip" (1, unrelated), "netty-http InputStream read negative" (1, unrelated); GitHub pull requests "NettyChannelBufferStreamCache": 1 merged (#25247, locks), none open.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)