You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "dfengliu (via GitHub)" <gi...@apache.org> on 2026/09/10 12:42:33 UTC

[PR] fix: validate size in FixedIndexed header (druid)

dfengliu opened a new pull request, #20315:
URL: https://github.com/apache/druid/pull/20315

   ### Description
   
   `FixedIndexed.read()` trusts the `size` header field. A negative value surfaces as a raw JVM exception at load (`IllegalArgumentException: newPosition < 0`), and a crafted size whose `width * size` arithmetic wraps is **silently accepted**: e.g. `size = 0x40000002` with `width = 8` claims 1073741826 elements while the buffer holds two longs, and `get(1073741824)` / `get(1073741825)` return the values of slot 0 / slot 1 because `valuesOffset + index * width` wraps — duplicated/shifted reads with no error.
   
   #### Fixed the bug ...
   
   Validate the header before any offset arithmetic: `size >= 0`, and `width * valuesCount <= buffer.remaining()` (computed in long to avoid the same wrap). Malformed headers now fail fast with a descriptive `IllegalArgumentException` (`"size[%s] must be non-negative"` / `"size[%s] with width[%s] exceeds the available buffer"`).
   
   Added `testNegativeSizeRejected` and `testSizeExceedingBufferRejected` to `FixedIndexedTest`. I verified both tests fail without the fix (reproducing the raw `newPosition < 0` / `newPosition > limit` exceptions) and pass with it; `mvn validate` (checkstyle) and `mvn -DstrictCompile test-compile` (Error Prone) also pass locally.
   
   #### Release note
   
   - Reject malformed `FixedIndexed` headers (negative size, or size exceeding the available buffer) with a descriptive error instead of failing later with an unrelated JVM exception or silently returning wrong values.
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `FixedIndexed`
    * `FixedIndexedTest`
   
   <hr>
   
   This PR has:
   
   - [x] been self-reviewed.
   - [ ] added documentation for new or modified features or behaviors.
   - [x] a release note entry in the PR description.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: validate size in FixedIndexed header (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20315:
URL: https://github.com/apache/druid/pull/20315#discussion_r3979690063


##########
processing/src/main/java/org/apache/druid/segment/data/FixedIndexed.java:
##########
@@ -63,6 +63,14 @@ public static <T> Supplier<FixedIndexed<T>> read(ByteBuffer bb, TypeStrategy<T>
     final boolean isSorted = (flags & IS_SORTED_MASK) == IS_SORTED_MASK ? true : false;
     Preconditions.checkState(!(hasNull && !isSorted), "cannot have null values if not sorted");
     final int size = buffer.getInt() + (hasNull ? 1 : 0);
+    Preconditions.checkArgument(size >= 0, "size[%s] must be non-negative", size);
+    final int valuesCount = hasNull ? size - 1 : size;

Review Comment:
   [P2] Reject a negative valuesCount after null adjustment
   
   When the null flag is set, the header count is incremented before this line. A crafted header with `hasNull` and a serialized count of `-1` therefore produces `size == 0` and `valuesCount == -1`; the following upper-bound check still passes because the byte count is negative. The final position update then moves `bb` backwards by `width` (for the width-4 dictionary this lands in the header), so callers that parse subsequent sections read them from the wrong offset, while other callers can construct a zero-sized dictionary despite the null flag. Reject a negative raw count or require `valuesCount >= 0` before the multiplication, and cover this null-flag case with a test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: validate size in FixedIndexed header (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 merged PR #20315:
URL: https://github.com/apache/druid/pull/20315


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] fix: validate size in FixedIndexed header (druid)

Posted by "dfengliu (via GitHub)" <gi...@apache.org>.
dfengliu commented on code in PR #20315:
URL: https://github.com/apache/druid/pull/20315#discussion_r3987667981


##########
processing/src/main/java/org/apache/druid/segment/data/FixedIndexed.java:
##########
@@ -63,6 +63,14 @@ public static <T> Supplier<FixedIndexed<T>> read(ByteBuffer bb, TypeStrategy<T>
     final boolean isSorted = (flags & IS_SORTED_MASK) == IS_SORTED_MASK ? true : false;
     Preconditions.checkState(!(hasNull && !isSorted), "cannot have null values if not sorted");
     final int size = buffer.getInt() + (hasNull ? 1 : 0);
+    Preconditions.checkArgument(size >= 0, "size[%s] must be non-negative", size);
+    final int valuesCount = hasNull ? size - 1 : size;

Review Comment:
   @FrankChen021 you are right. When the null flag is set the serialized count is incremented before validation, so a raw count of `-1` produced `size == 0` / `valuesCount == -1`: it passed both checks and moved the buffer position backwards. Fixed in 9c7e0983 — the check now validates `valuesCount` (the post-adjustment derived value), and I added `testNegativeCountWithNullFlagRejected` covering the null-flag case. I verified the new test fails without the fix (reproducing `newPosition < 0`) and passes with it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org