You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "prashanthbdremio (via GitHub)" <gi...@apache.org> on 2026/06/23 23:05:28 UTC

[I] Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity [arrow-java]

prashanthbdremio opened a new issue, #1194:
URL: https://github.com/apache/arrow-java/issues/1194

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   `ListVector` and `LargeListVector` can expose an invalid offset buffer state when `valueCount == 0`.
   
   For an empty list vector, the logical offset buffer should still contain the leading offset entry:
   
   - `ListVector`: `(valueCount + 1) * 4 == 4` bytes
   - `LargeListVector`: `(valueCount + 1) * 8 == 8` bytes
   
   However, in the empty-vector path, the offset buffer can have:
   
   ```text
   readerIndex: 0
   writerIndex: 4
   capacity: 0
   ```
   
   or the equivalent `writerIndex: 8, capacity: 0` for `LargeListVector`.
   
   This violates the normal buffer invariant:
   
   ```text
   0 <= readerIndex <= writerIndex <= capacity
   ```
   
   Downstream consumers that unwrap or serialize the Arrow buffer through Netty can then fail with:
   
   ```text
   IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
   (expected: 0 <= readerIndex <= writerIndex <= capacity(0))
   ```
   
   The issue is that `setReaderAndWriterIndex()` sets the offset buffer writer index based on `valueCount * OFFSET_WIDTH`, which is `0` for empty vectors. But list vectors still require one offset slot even when there are no values.
   
   The same issue applies to both:
   
   - `org.apache.arrow.vector.complex.ListVector`
   - `org.apache.arrow.vector.complex.LargeListVector`
   
   ### Expected behavior
   
   For `valueCount == 0`, the offset buffer should still have enough capacity and readable bytes for the leading zero offset:
   
   ```text
   (valueCount + 1) * OFFSET_WIDTH
   ```
   
   So:
   
   - empty `ListVector` should expose at least 4 bytes for offset `[0]`
   - empty `LargeListVector` should expose at least 8 bytes for offset `[0]`
   
   The first offset value should be zero.
   
   ### Actual behavior
   
   An empty list vector can expose an offset buffer with a non-zero writer index but zero capacity, causing Netty buffer validation to fail when the buffer is unwrapped or consumed.
   
   ### Suggested fix
   
   Update `ListVector.setReaderAndWriterIndex()` and `LargeListVector.setReaderAndWriterIndex()` so the offset buffer writer index is based on:
   
   ```java
   (valueCount + 1) * OFFSET_WIDTH
   ```
   
   For the `valueCount == 0` case, ensure the offset buffer has enough capacity for the leading zero offset before setting the writer index.
   
   Care should be taken not to shrink the vector's future offset allocation size when allocating this empty sentinel offset buffer.
   
   ### Additional context
   
   This was observed downstream in Dremio after upgrading Arrow Java. The failure occurred while sending a record batch containing an empty list vector, where the send path unwraps Arrow buffers through Netty.
   
   The downstream error was:
   
   ```text
   SYSTEM ERROR: IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
   (expected: 0 <= readerIndex <= writerIndex <= capacity(0))
   ```
   
   This issue is distinct from #1125. That issue involves `UnionListReader.setPosition` on a post-IPC empty list. This issue is about the offset buffer exported by empty `ListVector` / `LargeListVector` instances having an invalid writer-index/capacity relationship.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity [arrow-java]

Posted by "prashanthbdremio (via GitHub)" <gi...@apache.org>.
prashanthbdremio commented on issue #1194:
URL: https://github.com/apache/arrow-java/issues/1194#issuecomment-4812944808

   cc: @Yicong-Huang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity [arrow-java]

Posted by "jarohen (via GitHub)" <gi...@apache.org>.
jarohen commented on issue #1194:
URL: https://github.com/apache/arrow-java/issues/1194#issuecomment-6290660015

   Hey folks - we hit this in XTDB too, in a slightly worse form: when the empty list's offset buffer is allocator.getEmpty(), it's NettyAllocationManager.EMPTY_BUFFER, a single ArrowBuf shared across the whole JVM.
   
   `setReaderAndWriterIndex` then sets that shared buffer's writer index to 4, and every other vector holding the empty buffer sees it. In our case that's other threads' validity buffers mid-serialisation, which then fail with `IndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))` or go out with the wrong length. Under 8 concurrent Flight streams, roughly 1 in 7 batches came out corrupted; with one stream it never happens, because a later vector resets the index to 0 before anything serialises it.
   
   So whatever #1229 settles on for freshly constructed or cleared vectors, it'd be worth making sure the empty-vector path never sets a non-zero index on the allocator's empty buffer: allocate the 4-byte [0] instead, or leave the index alone when capacity is 0.
   
   See xtdb/xtdb#6231 for the write-up and workaround on our side.
   
   Cheers šŸ™‚ 
   
   James


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity [arrow-java]

Posted by "telemenar (via GitHub)" <gi...@apache.org>.
telemenar commented on issue #1194:
URL: https://github.com/apache/arrow-java/issues/1194#issuecomment-4858336810

   There is some nuance here because there are two things in tension:
   
   * Arrow layout/spec compliance: a list array with `valueCount == 0` still has one offset entry, so the exported offset buffer should contain `offset[0] == 0`.
   * Current `ListVector` / `LargeListVector` lifecycle behavior: freshly constructed, cleared, or otherwise empty vectors may legitimately have no allocated buffers yet.
   
   Either way, the current `setReaderAndWriterIndex()` behavior for a `valueCount == 0` `ListVector` is guaranteed to produce an invalid buffer state if the vector is still in the same state returned by `ListVector.empty()`.
   
   There are also a few ways to get into that state somewhat unexpectedly. For example, `ListVector.TransferImpl.splitAndTransfer()` calls `ListVector.clear()` on the destination vector. When the split length is `0`, that can release the destination buffers and leave the destination with `allocator.getEmpty()` as its offset buffer.
   
   Given the existing clear/reset behavior and the surrounding class hierarchy, I suspect persistent early enforcement would be tricky to implement cleanly. It would mean preserving or recreating the one-entry offset buffer across construction, `clear()`, and other empty-vector transitions, which may conflict with existing assumptions that an empty vector can hold no buffers.
   
   So boundary enforcement may be the more practical direction: allow the internal empty/unallocated state, but materialize the required zero offset at API boundaries where Arrow physical layout matters.
   
   The design questions I’m not fully sure about are:
   
   * Other than the paths that call `setReaderAndWriterIndex()`, are there other boundaries that need the same protection?
   * Is it acceptable for `getFieldBuffers()` to allocate the required one-entry offset buffer as part of preparing the vector for export/serialization?
   * Should call sites like zero-length `splitAndTransfer()` also enforce this before returning a schema-visible destination vector, or should that be left entirely to export-time handling?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org