You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/08/20 03:13:43 UTC

[I] NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue [arrow-dotnet]

CurtHagenlocher opened a new issue, #418:
URL: https://github.com/apache/arrow-dotnet/issues/418

   `NativeBuffer<TItem, TTracker>.Grow` doubles the current length in a `checked` context and does not
   saturate, so once a buffer passes half of the addressable maximum its **next** grow throws
   `OverflowException` — however small the requested increase, and even though the requested size still
   fits comfortably.
   
   ```csharp
   // src/Apache.Arrow/Memory/NativeBuffer.cs
   // Exponential growth (2x) to amortise repeated grows
   // TODO: There might be a size that's big enough to work for this case but not too big to overflow.
   // We could use that instead of blindly doubling.
   int newCount = Math.Max(newElementCount, checked(Length * 2));
   ```
   
   The existing `TODO` on those lines describes exactly this.
   
   For a `NativeBuffer<byte, …>` that means a hard ceiling near **1 GiB**: at `Length` above
   `int.MaxValue / 2`, `checked(Length * 2)` overflows, so a buffer cannot grow to 1.2 GiB even though
   `int.MaxValue` bytes are addressable and `Memory<byte>` could hold them.
   
   ## Impact
   
   Any caller that grows a buffer incrementally past that point fails, and fails misleadingly: the
   exception is arithmetic overflow, which reads as an internal error rather than as a size limit the
   caller was approaching. There is no way for a caller to avoid it — asking for a smaller increment
   does not help, because the overflow is in the doubling and not in the request.
   
   I hit this decoding a large Parquet BYTE_ARRAY column, where the buffer grows once per page. A column
   with roughly 1.5 GiB of string data — comfortably inside what a `StringArray` can address — failed
   with `OverflowException` from inside the allocator rather than reading successfully.
   
   ## Expected
   
   Growth saturates at the largest addressable element count instead of overflowing, so a request that
   fits succeeds. A request that genuinely cannot be addressed should still fail, as it does today, at
   the byte-size calculation.
   
   ## Note on the neighbouring limit
   
   This is separate from the 2 GiB ceiling on `ArrowBuffer` itself (`ReadOnlyMemory<byte>`, `int
   Length`). Fixing this does not raise that ceiling; it just stops buffers failing at half of it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher closed issue #418: NativeBuffer.Grow throws OverflowException instead of growing once the buffer passes half of int.MaxValue
URL: https://github.com/apache/arrow-dotnet/issues/418


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org