You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "gfeyer (via GitHub)" <gi...@apache.org> on 2026/02/08 17:39:24 UTC
[PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
gfeyer opened a new pull request, #3646:
URL: https://github.com/apache/avro/pull/3646
What is the purpose of the change
BinaryDecoder::arrayNext() calls doDecodeLong() directly instead of doDecodeItemCount(), causing it to mishandle negative array block counts. Per the Avro spec, a negative block count means the absolute value is the item count followed by an additional long for the byte-size of the block. When arrayNext() reads a negative count, static_cast<size_t>(-100) produces a huge value and the byte-size long is left unconsumed, corrupting the stream position.
doDecodeItemCount() already handles this correctly and is used by arrayStart(), mapStart(), and mapNext(). Only arrayNext() bypassed it. The fix changes arrayNext() to call doDecodeItemCount() for consistency.
This affects any array large enough to be encoded in multiple blocks with negative counts. ClickHouse independently found the same bug (https://github.com/ClickHouse/ClickHouse/issues/60438, https://github.com/ClickHouse/avro/pull/23).
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
The fix was verified against production Avro messages containing arrays with 266+ items encoded in multiple blocks with negative counts, which previously failed ~20% of the time and now decode correctly.
Documentation
- Does this pull request introduce a new feature? no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on PR #3646:
URL: https://github.com/apache/avro/pull/3646#issuecomment-4944222213
@RyanSkraba this one has to be cherry picked into the next 1.12 release too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #3646:
URL: https://github.com/apache/avro/pull/3646#issuecomment-3907823540
Thank you, @gfeyer !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #3646:
URL: https://github.com/apache/avro/pull/3646#discussion_r2780921649
##########
lang/c++/test/CodecTests.cc:
##########
@@ -2100,6 +2100,42 @@ static void testJsonCodecReinit() {
}
}
+static void testArrayNegativeBlockCount() {
Review Comment:
This test pass even without the fix above.
The reason is that the negative count is in the first block and this uses `arrayStart()` which already delegates to `doDecodeItemCount()`.
Please update the test to use a negative count [also] in the second block.
##########
lang/c++/impl/BinaryDecoder.cc:
##########
Review Comment:
This may lead to overflow if `result` is size_t's min value.
Possible improvement:
```suggestion
return static_cast<size_t>(-(result + 1)) + 1;
```
##########
lang/c++/test/CodecTests.cc:
##########
@@ -2100,6 +2100,42 @@ static void testJsonCodecReinit() {
}
}
+static void testArrayNegativeBlockCount() {
+ // Array of ints [10, 20, 30, 40, 50] encoded with a negative block count.
+ // Per the Avro spec, a negative count means: abs(count) items follow,
+ // preceded by a long byte-size of the block.
+ //
+ // Block 1: count=-3, bytesize=3, items: 10, 20, 30
+ // Block 2: count=2, items: 40, 50
+ // Terminal: count=0
+ const uint8_t data[] = {
+ 0x05, // zigzag(-3) = 5
+ 0x06, // zigzag(3) = 6 (byte-size of block)
+ 0x14, 0x28, 0x3c, // zigzag ints: 10, 20, 30
+ 0x04, // zigzag(2) = 4
+ 0x50, 0x64, // zigzag ints: 40, 50
+ 0x00 // terminal
+ };
+
+ InputStreamPtr is = memoryInputStream(data, sizeof(data));
+ DecoderPtr d = binaryDecoder();
+ d->init(*is);
+
+ std::vector<int32_t> result;
+ for (size_t n = d->arrayStart(); n != 0; n = d->arrayNext()) {
+ for (size_t i = 0; i < n; ++i) {
+ result.push_back(d->decodeInt());
+ }
+ }
+
+ BOOST_CHECK_EQUAL(result.size(), 5u);
+ BOOST_CHECK_EQUAL(result[0], 10);
+ BOOST_CHECK_EQUAL(result[1], 20);
+ BOOST_CHECK_EQUAL(result[2], 30);
+ BOOST_CHECK_EQUAL(result[3], 40);
+ BOOST_CHECK_EQUAL(result[4], 50);
Review Comment:
```suggestion
const std::vector<int32_t> expected = {10, 20, 30, 40, 50};
BOOST_CHECK_EQUAL_COLLECTIONS(result.begin(), result.end(), expected.begin(), expected.end());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "gfeyer (via GitHub)" <gi...@apache.org>.
gfeyer commented on code in PR #3646:
URL: https://github.com/apache/avro/pull/3646#discussion_r2782673369
##########
lang/c++/impl/BinaryDecoder.cc:
##########
Review Comment:
In practice this would mean an array with 2^63 items? Quite large for a prod system in my view, but technically it's undefined behavior so you're right, needs fixing, good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on code in PR #3646:
URL: https://github.com/apache/avro/pull/3646#discussion_r2786074874
##########
lang/c++/impl/BinaryDecoder.cc:
##########
Review Comment:
I don't expect that such big number will be used in normal conditions!
But an attacker can craft it manually and pass it to a system to cause problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] AVRO-4228: [c++] Fix BinaryDecoder::arrayNext() to handle negative block counts [avro]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g merged PR #3646:
URL: https://github.com/apache/avro/pull/3646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@avro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org