[GitHub] [arrow] Kimahriman opened a new issue, #34046: [Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces

["Kimahriman (via GitHub)" <gi...@apache.org>]

Kimahriman opened a new issue, #34046:
URL: https://github.com/apache/arrow/issues/34046

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   https://github.com/apache/arrow/pull/13815 fixed an issue with a misleading error caused by going over the 2GiB limit of a variable width vector, but it also introduced a 1 GiB limit when using the safe interfaces. Basically whenever you try to add data beyond 1 GiB, the vector will try to double itself to the next power of two, which would be 2147483648, which is greater than Integer.MAX_VALUE which is 2147483647, thus throwing a OversizedAllocationException. This effectively limits the total size allowed in the buffer to 1 GiB, even though it should only really be bound by the max offset being less than Integer.MAX_VALUE. 
   
   See https://github.com/apache/spark/pull/39572#issuecomment-1383195213 and the comment above it for how I could recreate the issue.
   
   ### Component(s)
   
   Java


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jjbskir commented on issue #34046: [Java] BaseVariableWidthVector only supports 1 GiB through safe interfaces

["jjbskir (via GitHub)" <gi...@apache.org>]

jjbskir commented on issue #34046:
URL: https://github.com/apache/arrow/issues/34046#issuecomment-1660582514

   Hey! I ran into the same issue recently. I wanted to see if there was any progress to support variables between 1GB to 2GB? Or any supported work arounds? 
   
   Happy to open a PR to support this. I think an solution would to add an additional check in [reallocDataBuffer](https://github.com/apache/arrow/blob/main/java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L539 ) if the `desiredAllocSize` is greater then 1GB, but less then 2GB to just set the `newAllocationSize` to the max value.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org