You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/08/31 03:19:12 UTC

[GitHub] [arrow] PeterSansan opened a new issue #8081: Is't possible to create ValueVector larger than Integer.Max byte limit in java

PeterSansan opened a new issue #8081:
URL: https://github.com/apache/arrow/issues/8081


   Hi. I use Float8Vector to store data which stand for relationship table with consecutive addresses in java version. If I want to build a 1000 fields * 150000 rows' relationship table which reshapes 1.5*10^8 rows' Float8Vector, it causes error like this:
   ```
   java.lang.IllegalArgumentException: the requested size must be non-negative
   
   	at org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:136)
   	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:292)
   	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:281)
   	at org.apache.arrow.vector.BaseValueVector.allocFixedDataAndValidityBufs(BaseValueVector.java:194)
   	at org.apache.arrow.vector.BaseFixedWidthVector.allocateBytes(BaseFixedWidthVector.java:329)
   	at org.apache.arrow.vector.BaseFixedWidthVector.allocateNew(BaseFixedWidthVector.java:299)
   	at org.apache.arrow.vector.BaseFixedWidthVector.allocateNew(BaseFixedWidthVector.java:264)
   ```
   I use the api are "setInitialCapacity" and "allocateNew". So I ask it's possible to create a Float8Vector larger than Integer.Max byte,  or does arrow have a data structure to stand for a relationship table with consecutive addresses. Consecutive addresses are important to me, because can transfer it to the other Computing system with the same memory.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield closed issue #8081: Is't possible to create ValueVector larger than Integer.Max byte limit in java

Posted by GitBox <gi...@apache.org>.

emkornfield closed issue #8081:
URL: https://github.com/apache/arrow/issues/8081


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on issue #8081: Is't possible to create ValueVector larger than Integer.Max byte limit in java

Posted by GitBox <gi...@apache.org>.

emkornfield commented on issue #8081:
URL: https://github.com/apache/arrow/issues/8081#issuecomment-684176420


   @PeterSansan this should be possible as of 1.0.1.  You will need to ensure you use the [Unsafe memory allocator](https://github.com/apache/arrow/blob/2092e18752a9c0494799493b12eb1830052217a2/java/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java)  if you have further issues opening a JIRA or writing an email to user@ or dev@ mailing lists is how we prefer to provide support.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org