You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lidavidm (via GitHub)" <gi...@apache.org> on 2026/09/10 06:29:16 UTC
[PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
lidavidm opened a new pull request, #1291:
URL: https://github.com/apache/arrow-java/pull/1291
Also, document that when enable_unsafe_memory_access is enabled, all bets are off.
Reported by n0mi1k.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#discussion_r3985661522
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
@@ -1526,19 +1518,43 @@ public int hashCode(int index, ArrowBufHasher hasher) {
if (isNull(index)) {
return ArrowBufPointer.NULL_HASH_CODE;
}
- final int length = getValueLength(index);
- if (length < INLINE_SIZE) {
- int start = index * ELEMENT_SIZE + LENGTH_WIDTH;
- return ByteFunctionHelpers.hash(hasher, this.getDataBuffer(), start, start + length);
- } else {
- final int bufIndex =
+ return getData(
+ index,
+ (buf, offset, length) -> ByteFunctionHelpers.hash(hasher, buf, offset, offset + length));
+ }
+
+ @FunctionalInterface
+ protected interface ViewElementConsumer<T> {
+ T consume(ArrowBuf buf, int offset, int length);
+ }
+
+ /** Helper to get a single view value with sanity checking. */
+ protected <T> T getData(int index, ViewElementConsumer<T> consumer) {
Review Comment:
The new checked access path does not cover `splitAndTransferViewBufferAndDataBuffer`. Its out-of-line branch still calls `target.allocateOrGetLastDataBuffer(stringLength)` before validating the source offset/length, so a crafted IPC view can still trigger the large allocation this change is intended to prevent via `splitAndTransferTo`. Route that branch through `getData` (or apply the same validation) before allocating the target buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on code in PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#discussion_r3979282085
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
I'm a bit surprised as #1217 names this method as affected, but it isn't touched by this PR.
`buffer.set(dataBuf, dataOffset, dataLength)` -> `ReusableByteArray.set()` still does `bytes = new byte[len]` before the source `getBytes()` bounds check runs. The same allocate-before-check pattern just fixed two methods above in `getData(int)`.
Should we include the same change here?
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
Named in #1217 😄
`target.allocateOrGetLastDataBuffer(stringLength)` allocated before `currentDataBuf.setBytes(currentOffset, dataBuf, readBufOffset, stringLength)` bounds-checks the source read, and `stringLength` (from `getValueLength(i)`) is never validated first.
This runs on every split/slice/transfer of a View vector, so it's a fairly reachable path for the same bug class this PR addresses elsewhere.
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
Same issue as `getDataPointer`: `dataOffset`/`bufIndex` feed `ByteFunctionHelpers.hash(hasher, dataBuf, dataOffset, dataOffset + length)` with no validation.
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
Named in #1217 😄
No array allocation here, but `bufIndex`/`dataOffset` from the view record are used to build an `ArrowBufPointer` with zero validation. Under `arrow.enable_unsafe_memory_access=true`, this is the "reads arbitrary native heap" case, not just a DoS.
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
Same here: it's mentioned in #1217 but not changed in this PR.
`allocateOrGetLastDataBuffer(viewLength)` allocates a buffer sized by `viewLength` before the bounds-checked `thisDataBuf.setBytes(..., dataBuf, dataOffset, viewLength)` ever runs. A corrupted source view (let's say `copyFrom`/`copyFromSafe` on data loaded from an untrusted IPC stream) drives the same large-allocation DoS this PR fixes in `getData`, through the copy path instead.
##########
vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java:
##########
@@ -2921,4 +2921,20 @@ public void testValidate() {
assertTrue(e.getMessage().contains("Not enough capacity for data buffer"));
}
}
+
+ @Test
+ public void testValidateInvalidOffsets() {
Review Comment:
`ArrowBuf.setInt(index, value)` takes a byte offset, not a field-slot index. The view record layout is length@0, prefix@4, bufferIndex@8, dataOffset@12 (4 bytes each), but `setInt(1, 0)`, `setInt(2, 0)`, `setInt(3, 1024)` write to byte offsets 1/2/3, overlapping the length field instead of landing on prefix/bufferIndex/dataOffset.
I just tested it: length ends up 64 (unchanged), prefix=4 (collateral overlap), bufferIndex=0, dataOffset=0 (not the intended 1024).
The test passes, but I believe because dataLength=64 exceeds the tiny data buffer's capacity, not because of an out-of-range dataOffset like the test name implies.
I suggest to use `setInt(12, 1024)` to actually cover the corrupted-offset scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#discussion_r3985412029
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
Hmm, adding manual checks everywhere is going to be fairly brittle...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#discussion_r3986795949
##########
vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java:
##########
@@ -2921,4 +2921,21 @@ public void testValidate() {
assertTrue(e.getMessage().contains("Not enough capacity for data buffer"));
}
}
+
+ @Test
+ public void testValidateInvalidOffsets() {
+ try (final ViewVarCharVector vector = new ViewVarCharVector("v", allocator)) {
+ vector.allocateNew(16, 1);
+ vector.allocateOrGetLastDataBuffer(8);
+ var offsets = vector.getDataBuffer();
+ offsets.setInt(0, 64);
Review Comment:
This value is too small to regress the allocation-order bug: the previous implementation would allocate a 64-byte array and then throw the same `IndexOutOfBoundsException` from `getBytes`, so this test also passes before the fix. Use a length that cannot be allocated (or otherwise observe allocation) so the test proves the invalid range is rejected before allocation; `Integer.MAX_VALUE` exercises the reported failure while the new explicit bounds check rejects it first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#issuecomment-5986232599
Rebased, CI is green
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#discussion_r3985650536
##########
vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java:
##########
Review Comment:
I consolidated all of these methods to use a single accessor that always checks the buffer size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#issuecomment-5614233947
<!-- label_helper_comment -->
Thank you for opening a pull request!
Please label the PR with one or more of:
- bug-fix
- chore
- dependencies
- documentation
- enhancement
Also, add the 'breaking-change' label if appropriate.
See [CONTRIBUTING.md](https://github.com/apache/arrow-java/blob/main/CONTRIBUTING.md) for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-1217: Bounds check before alloc in var-width view vectors [arrow-java]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #1291:
URL: https://github.com/apache/arrow-java/pull/1291#issuecomment-5629873445
@lidavidm thanks for the updates, I'm doing a new pass but it looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org