You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Sriniketh24 (via GitHub)" <gi...@apache.org> on 2026/05/23 02:39:12 UTC
[PR] GH-36388: [C++][Python] Return error from MakeArrayFromScalar on offset overflow [arrow]
Sriniketh24 opened a new pull request, #50024:
URL: https://github.com/apache/arrow/pull/50024
### Rationale
`pyarrow.repeat` (backed by `MakeArrayFromScalar` in C++) silently created an invalid array with negative offsets when the total data size (`value_size * repetition_count`) exceeded `INT32_MAX` for 32-bit offset types (`StringType`, `BinaryType`). The resulting array passed creation without error but failed validation with a cryptic "Negative offsets in binary array" or "non-monotonic offset" message.
### What changed
Added an early overflow check in `RepeatedArrayFactory::CreateOffsetsBuffer` that computes the total data size in `int64_t` and returns `Status::Invalid` with an actionable error message when it would exceed the offset type's maximum. The error message suggests using `large_*` types (e.g. `large_string`, `large_binary`) for data exceeding 2 GB.
### Are these changes tested?
Yes.
- **C++ test**: `TestMakeArrayFromScalarOffsetOverflow` in `array_test.cc` — tests string, binary, and large_string scalars
- **Python test**: `test_repeat_offset_overflow` in `test_array.py` — verifies `pa.repeat` raises `ArrowInvalid` on overflow
### Are there any user-facing changes?
Yes. `MakeArrayFromScalar` (and `pyarrow.repeat`) now raises `ArrowInvalid` early with a clear error message instead of silently returning a corrupt array. This is a strictly better user experience.
Closes: #36388
---
This is AI-assisted work by Claude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36388: [C++][Python] Return error from MakeArrayFromScalar on offset overflow [arrow]
Posted by "Sriniketh24 (via GitHub)" <gi...@apache.org>.
Sriniketh24 commented on PR #50024:
URL: https://github.com/apache/arrow/pull/50024#issuecomment-5531163621
Thanks for the pointer @AlenkaF — that discussion was exactly the missing piece. @bkietz caught a real bug in the approach I'd pushed: checking `length_` alone against `OffsetType::max()` is a false positive — an empty string repeated more than `int32::max()` times is perfectly valid (every offset stays 0), so that check would have incorrectly rejected it.
Applied bkietz's suggested fix directly: compute `value_length * length_` in `int64_t` and compare the product against `OffsetType::max()`, rather than checking either operand individually.
Also added the boundary tests js8544 asked for (empty string at length > int32::max, and "aa" at exactly int32::max/2) — these need multi-GB allocations to actually succeed rather than fail-fast, so I gated them behind `LARGE_MEMORY_TEST` like the rest of the codebase does (e.g. `table_test.cc`) rather than adding that cost to every CI run.
Local syntax check (now with gmock available) is clean on both files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36388: [C++][Python] Return error from MakeArrayFromScalar on offset overflow [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50024:
URL: https://github.com/apache/arrow/pull/50024#discussion_r3299910011
##########
cpp/src/arrow/array/util.cc:
##########
@@ -853,6 +853,19 @@ class RepeatedArrayFactory {
template <typename OffsetType>
Status CreateOffsetsBuffer(OffsetType value_length, std::shared_ptr<Buffer>* out) {
+ // Check that the total data size does not overflow the offset type.
+ // For 32-bit offset types (e.g. StringType, BinaryType), value_length * length_
+ // must fit in int32_t, otherwise the offsets wrap around and produce an invalid
+ // array with negative offsets.
+ if (value_length > 0 && length_ > 0) {
+ int64_t total_size = static_cast<int64_t>(value_length) * length_;
+ if (total_size > static_cast<int64_t>(std::numeric_limits<OffsetType>::max())) {
+ return Status::Invalid(
+ "Cannot create array: total data size (", total_size,
Review Comment:
The overflow check computes `total_size` via `static_cast<int64_t>(value_length) * length_`, which can overflow `int64_t` (signed overflow is UB) for large `length_`. Also, the `for (..., offset += value_length)` loop increments `offset` once after the last appended offset, which can overflow `OffsetType` even when the final stored offset is within range. Both issues can be avoided by using a division-based overflow check and by appending the final offset without an extra increment.
##########
cpp/src/arrow/array/util.cc:
##########
@@ -853,6 +853,19 @@ class RepeatedArrayFactory {
template <typename OffsetType>
Status CreateOffsetsBuffer(OffsetType value_length, std::shared_ptr<Buffer>* out) {
+ // Check that the total data size does not overflow the offset type.
+ // For 32-bit offset types (e.g. StringType, BinaryType), value_length * length_
+ // must fit in int32_t, otherwise the offsets wrap around and produce an invalid
+ // array with negative offsets.
+ if (value_length > 0 && length_ > 0) {
Review Comment:
`CreateOffsetsBuffer()` is called *after* `CreateBufferOf()` in the var-binary / string repeat path (and similarly for some nested types). For the overflow cases this PR targets (e.g. repeating a 1-byte string 2^31 times), `CreateBufferOf()` would attempt a huge `Resize()` and then loop `length_` times appending bytes before the offset overflow is detected, which is effectively non-terminating / can OOM. To make the new overflow check effective and ensure the intended `Status::Invalid` is returned quickly, the offset-size check needs to run before building the repeated values buffer (e.g. compute/check offsets first, then allocate/copy values).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36388: [C++][Python] Return error from MakeArrayFromScalar on offset overflow [arrow]
Posted by "Sriniketh24 (via GitHub)" <gi...@apache.org>.
Sriniketh24 commented on PR #50024:
URL: https://github.com/apache/arrow/pull/50024#issuecomment-5243191020
Hi @AlenkaF — thanks for the pointer to #38504, that was a good approach that just ran out of the original author's time (credit to @llama90 for the initial work, and @js8544 for reviewing it there).
Reworked this PR to build on that approach:
- Swapped the manual `int64_t` multiplication for `arrow::internal::MultiplyWithOverflow`, since the previous approach could itself silently overflow for 64-bit offset types (large_string/large_binary) — using the type-generic overflow-safe helper avoids that.
- Added the length-exceeds-offset-type check and the negative-length check at `MakeArrayFromScalar`, matching what was validated in #38504.
- Kept the existing Python-level `pa.repeat()` regression test from this PR as additional coverage beyond the C++-only tests in #38504.
Note: I wasn't able to run the full C++ test suite in my local environment (no existing build directory / ninja), so I only syntax-checked the changed file directly against the local headers (clean, no errors). CI here will be the real validation — flagging that up front rather than claiming more confidence than I have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36388: [C++][Python] Return error from MakeArrayFromScalar on offset overflow [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on PR #50024:
URL: https://github.com/apache/arrow/pull/50024#issuecomment-5343646102
Could you have a look at this comment from the linked PR: https://github.com/apache/arrow/pull/38504#discussion_r1394400239 and let me know what you think of the suggestion?
> Note: I wasn't able to run the full C++ test suite in my local environment (no existing build directory / ninja), so I only syntax-checked the changed file directly against the local headers (clean, no errors). CI here will be the real validation — flagging that up front rather than claiming more confidence than I have.
And please, don't use agents to communicate with us. It is OK to use the tools for development provided you understand the changes. Also, you could point your agent to the development docs so it can help with building from source and testing: https://arrow.apache.org/docs/developers/python/building.html#build-pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org