You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "vigneshsiva11 (via GitHub)" <gi...@apache.org> on 2026/02/05 17:19:19 UTC
[PR] Parquet: prevent binary offset overflow by stopping batch early [arrow-rs]
vigneshsiva11 opened a new pull request, #9362:
URL: https://github.com/apache/arrow-rs/pull/9362
# Which issue does this PR close?
- Closes #7973.
# Rationale for this change
When reading Parquet files containing very large binary or string values, the Arrow Parquet reader can attempt to construct a RecordBatch whose total value buffer exceeds the maximum representable offset size. This can lead to an overflow error or panic during decoding.
Instead of allowing the buffer to overflow and failing late, the reader should detect this condition early and stop decoding before the offset exceeds the representable limit. This behavior is consistent with other Arrow implementations (for example, PyArrow), which emit smaller batches when encountering very large row groups.
# What changes are included in this PR?
- Add an early overflow check when appending binary values to the Arrow offset buffer.
- Ensure the overflow condition is detected before mutating internal buffers.
- Return a controlled error instead of panicking when the offset limit would be exceeded.
- Apply the fix uniformly across all byte array decoding paths (plain, dictionary, and delta encodings) via the shared offset buffer logic.
# Are these changes tested?
Yes.
- Regression tests covering large binary values were added in a separate PR.
- Existing Parquet reader and writer tests continue to pass in CI.
Note: Some Parquet and Arrow integration tests require external test data provided via git submodules (parquet-testing and testing). These submodules are not present in a minimal local checkout but are initialized in CI.
# Are there any user-facing changes?
Yes.
- Reading Parquet files with very large binary or string columns will no longer panic or fail late due to offset overflow.
- The reader now stops batch construction early and reports the error safely.
There are no breaking changes to public APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: prevent binary offset overflow by stopping batch early [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on PR #9362:
URL: https://github.com/apache/arrow-rs/pull/9362#issuecomment-3909469984
Hi @alamb,
You're absolutely right - this PR doesn't actually implement batch splitting as described. After further investigation, I realized that:
**This PR (#9362) only:**
- Moves the overflow check to happen BEFORE buffer mutation - Prevents buffer corruption by detecting overflow early - Returns a clean error instead of potentially panicking
**This PR does NOT:**
- Emit smaller batches
- Stop decoding early to avoid overflow
- Change the batch size
**The complete solution is in PR #9369:**
https://github.com/apache/arrow-rs/pull/9369
PR #9369 includes the same defensive overflow check from this PR, PLUS the full implementation that actually splits RecordBatches when binary offsets would overflow. It properly detects the overflow condition during decoding and stops early, emitting smaller batches as described.
I'll close this PR in favor of #9369, which is the comprehensive solution that addresses issue #7973 as intended.
Thank you for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: prevent binary offset overflow by stopping batch early [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9362:
URL: https://github.com/apache/arrow-rs/pull/9362#issuecomment-3897640282
> Instead of allowing the buffer to overflow and failing late, the reader should detect this condition early and stop decoding before the offset exceeds the representable limit. This behavior is consistent with other Arrow implementations (for example, PyArrow), which emit smaller batches when encountering very large row groups.
This PR doesn't change the size of the batches emitted I don't think 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: prevent binary offset overflow by stopping batch early [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9362:
URL: https://github.com/apache/arrow-rs/pull/9362#issuecomment-3897637889
> Closes https://github.com/apache/arrow-rs/issues/7973.
I am not sure this PR closes the issue -- maybe it changes the error message? Or does it prevent a panic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Parquet: prevent binary offset overflow by stopping batch early [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 closed pull request #9362: Parquet: prevent binary offset overflow by stopping batch early
URL: https://github.com/apache/arrow-rs/pull/9362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org