You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "gszadovszky (via GitHub)" <gi...@apache.org> on 2025/04/22 08:43:53 UTC
[I] Allow using 1GB+ data buffers in variable width vectors [arrow-java]
gszadovszky opened a new issue, #721:
URL: https://github.com/apache/arrow-java/issues/721
### Describe the enhancement requested
In `BaseVariableWidthVector` the reallocation logic of the data buffer is to always use the "next power of 2" size instead of the actual desired size (see [here](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L574)). The reallocation fails if the new size exceeds the `MAX_BUFFER_SIZE` which is `Integer.MAX_VALUE` by default.
The issue is `Integer.MAX_VALUE` is just 1 byte less than a power of 2 (2<sup>31</sup> - 1) which means that exceeding the desired capacity of `1073741824` (1GB), the reallocation would fail. It means, we are unable to get a data buffer larger than 1GB while the default maximum size is almost 2GB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using 1GB+ data buffers in variable width vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm closed issue #721: Allow using 1GB+ data buffers in variable width vectors
URL: https://github.com/apache/arrow-java/issues/721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Allow using 1GB+ data buffers in variable width vectors [arrow-java]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #721:
URL: https://github.com/apache/arrow-java/issues/721#issuecomment-2820614216
We should clamp the size at 2GB, and we could consider a way to control the preallocation/reallocation behavior (e.g. to start with larger buffers up front)
In general we should be moving towards not having the 2GB limit, but it is baked in for now...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org