You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sachinnn99 (via GitHub)" <gi...@apache.org> on 2026/04/30 04:56:42 UTC
[PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
sachinnn99 opened a new pull request, #16167:
URL: https://github.com/apache/iceberg/pull/16167
Fixes #16127.
`SingleBufferInputStream.read()` and `MultiBufferInputStream.read()` throw `EOFException` when the stream is exhausted. This violates the `java.io.InputStream` contract, which requires the no-arg `read()` to return `-1` at EOF.
The multi-byte `read(byte[], int, int)` in both classes already correctly returns `-1` at EOF — the two overloads were inconsistent.
**Changes:**
- `SingleBufferInputStream.read()`: return `-1` instead of throwing `EOFException`
- `MultiBufferInputStream.read()`: return `-1` at both EOF entry points instead of throwing `EOFException`
- Update `testReadByte()` to assert `-1` return (including idempotency check)
Other `EOFException`-throwing methods (`slice()`, `sliceBuffers()`, `skipFully()`) are unchanged — they request specific byte counts where `EOFException` is the correct signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#issuecomment-4351253106
Thanks for the review! I looked at `slice()` — its `EOFException` is intentional since it's a "read exactly N bytes" operation (not part of the `InputStream` contract). Callers expect either the full slice or a failure, so throwing there is the correct behavior.
The fix here is scoped to the no-arg `read()`, which has a clear contract violation per `InputStream.read()` javadoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#issuecomment-4466494726
Thanks for the thorough review @laskoviymishka!
**On the 4 call sites:** Those methods use `org.apache.parquet.bytes.ByteBufferInputStream` (Parquet's class), not `org.apache.iceberg.io.ByteBufferInputStream` (the class modified here). They're in completely separate type hierarchies -- Parquet's extends `InputStream` directly, while Iceberg's extends `SeekableInputStream`. You can verify by checking the imports at line 24 in both `ValuesAsBytesReader.java` and `BaseVectorizedParquetValuesReader.java`. I also confirmed that no production code outside `core/io` references Iceberg's `ByteBufferInputStream`, so there are no callers that relied on the `EOFException` behavior.
**On unused imports:** `EOFException` is still actively used in `SingleBufferInputStream` (`seek()`, `slice()`, `sliceBuffers()`), `MultiBufferInputStream` (same methods + a catch clause), and the test file (`testWholeSliceBuffers`, `testSkipFully`). The imports are still needed.
**On enhancing `assertAtEOF`:** Good suggestion -- added `read(byte[])` assertion to the helper.
**On the drained multi-buffer test:** Added `testDrainedMultiBufferStream` which creates a multi-buffer stream from non-empty buffers, drains it via `read(byte[])`, then calls `assertAtEOF`. This explicitly exercises the `nextBuffer() -> return -1` path at line 275.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#issuecomment-4831913477
> Should we raise this upstream in parquet-java?
I think you should
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "sachinnn99 (via GitHub)" <gi...@apache.org>.
sachinnn99 commented on PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#issuecomment-4403683739
Thanks for the suggestions @Baunsgaard! Added in the latest push:
- `assertAtEOF` helper matching your snippet
- Called it after EOF is first encountered in `testReadAll`, `testSmallReads`, `testPartialBufferReads`, and `testReadByte`
- Added `testEmptyStream` covering single empty buffer, multiple empty buffers, and an empty buffer list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#discussion_r3249629785
##########
core/src/main/java/org/apache/iceberg/io/MultiBufferInputStream.java:
##########
@@ -263,7 +263,7 @@ public int read(byte[] bytes) {
@Override
public int read() throws IOException {
if (current == null) {
- throw new EOFException();
+ return -1;
Review Comment:
The contract fix is right, but flagging this for visibility: I found four existing call sites that consume the result of `read()` without a `-1` guard and currently depend on the throw to fail loudly. After this change they fail silently — or hang, in one case:
- `ValuesAsBytesReader.getByte()` (parquet) — casts to byte, `(byte) -1 = 0xFF` → 8 spurious true bits in boolean decoding
- `BaseVectorizedParquetValuesReader.readUnsignedVarInt()` (arrow) — `-1 & 0x80 != 0` always true → **infinite loop**
- `BaseVectorizedParquetValuesReader.readIntLittleEndian()` and `readIntLittleEndianPaddedOnBitWidth()` (arrow) — return garbage int instead of throwing
These callers shouldn't reach EOF on well-formed input (Parquet decoders know exact byte counts), so this is defense-in-depth, not a happy-path regression. But the failure mode changes from loud to silent/hanging.
Leaving the call to you — see options A/B/C in the top-level body. If you go with A (fix at callers), happy to look at the follow-up changes. Whatever you decide, could you note it in the PR description so it's visible to other reviewers?
##########
core/src/test/java/org/apache/iceberg/io/TestByteBufferInputStreams.java:
##########
@@ -35,6 +35,14 @@ public abstract class TestByteBufferInputStreams {
protected abstract void checkOriginalData();
+ private static void assertAtEOF(ByteBufferInputStream stream) throws IOException {
Review Comment:
Nice helper — the idempotency check is exactly the regression catch we want. One small addition to consider: assert `read(byte[])` returns `-1` too, so the helper pins both overloads at a single site. The multi-byte path already returns `-1` today, but bundling it means a future change to either overload trips the same assertion.
##########
core/src/main/java/org/apache/iceberg/io/SingleBufferInputStream.java:
##########
@@ -58,7 +58,7 @@ public long getPos() {
@Override
public int read() throws IOException {
if (!buffer.hasRemaining()) {
- throw new EOFException();
+ return -1;
Review Comment:
Minor: the `import java.io.EOFException;` at the top of this file is likely unused now (no `throw new EOFException()` remains). Same check for `MultiBufferInputStream.java` and for `import java.io.EOFException` in `TestByteBufferInputStreams.java` — the `assertThatThrownBy(...EOFException.class)` assertion is gone. Please drop the unused imports.
##########
core/src/test/java/org/apache/iceberg/io/TestByteBufferInputStreams.java:
##########
@@ -532,4 +540,11 @@ public void testMarkDoubleReset() throws Exception {
.isInstanceOf(IOException.class)
.hasMessageStartingWith("No mark defined");
}
+
+ @Test
+ public void testEmptyStream() throws Exception {
+ assertAtEOF(ByteBufferInputStream.wrap(ByteBuffer.allocate(0)));
+ assertAtEOF(ByteBufferInputStream.wrap(ByteBuffer.allocate(0), ByteBuffer.allocate(0)));
Review Comment:
Good coverage of the three construction shapes. Worth one more: a `MultiBufferInputStream` built from a non-empty buffer list, then fully drained — that exercises the `nextBuffer()` → `return -1` path at line 275, which is a structurally distinct branch from the `current == null` path at line 266. `testReadByte()` covers it for one shape, but a tiny dedicated drained-multi-buffer test pins the second branch explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #16167:
URL: https://github.com/apache/iceberg/pull/16167#issuecomment-4462849174
great reviewing @laskoviymishka
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix ByteBufferInputStream.read() to return -1 at EOF [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #16167:
URL: https://github.com/apache/iceberg/pull/16167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org