You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/08/20 03:13:43 UTC
[I] NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue [arrow-dotnet]
CurtHagenlocher opened a new issue, #418:
URL: https://github.com/apache/arrow-dotnet/issues/418
`NativeBuffer<TItem, TTracker>.Grow` doubles the current length in a `checked` context and does not
saturate, so once a buffer passes half of the addressable maximum its **next** grow throws
`OverflowException` — however small the requested increase, and even though the requested size still
fits comfortably.
```csharp
// src/Apache.Arrow/Memory/NativeBuffer.cs
// Exponential growth (2x) to amortise repeated grows
// TODO: There might be a size that's big enough to work for this case but not too big to overflow.
// We could use that instead of blindly doubling.
int newCount = Math.Max(newElementCount, checked(Length * 2));
```
The existing `TODO` on those lines describes exactly this.
For a `NativeBuffer<byte, …>` that means a hard ceiling near **1 GiB**: at `Length` above
`int.MaxValue / 2`, `checked(Length * 2)` overflows, so a buffer cannot grow to 1.2 GiB even though
`int.MaxValue` bytes are addressable and `Memory<byte>` could hold them.
## Impact
Any caller that grows a buffer incrementally past that point fails, and fails misleadingly: the
exception is arithmetic overflow, which reads as an internal error rather than as a size limit the
caller was approaching. There is no way for a caller to avoid it — asking for a smaller increment
does not help, because the overflow is in the doubling and not in the request.
I hit this decoding a large Parquet BYTE_ARRAY column, where the buffer grows once per page. A column
with roughly 1.5 GiB of string data — comfortably inside what a `StringArray` can address — failed
with `OverflowException` from inside the allocator rather than reading successfully.
## Expected
Growth saturates at the largest addressable element count instead of overflowing, so a request that
fits succeeds. A request that genuinely cannot be addressed should still fail, as it does today, at
the byte-size calculation.
## Note on the neighbouring limit
This is separate from the 2 GiB ceiling on `ArrowBuffer` itself (`ReadOnlyMemory<byte>`, `int
Length`). Fixing this does not raise that ceiling; it just stops buffers failing at half of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue [arrow-dotnet]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher closed issue #418: NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue
URL: https://github.com/apache/arrow-dotnet/issues/418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org