[jira] [Created] (ARROW-7746) [Java] Support large buffer for IPC

["Liya Fan (Jira)" <ji...@apache.org>]

Liya Fan created ARROW-7746:
-------------------------------

             Summary: [Java] Support large buffer for IPC
                 Key: ARROW-7746
                 URL: https://issues.apache.org/jira/browse/ARROW-7746
             Project: Apache Arrow
          Issue Type: Task
          Components: Java
            Reporter: Liya Fan


The motivation is described in https://github.com/apache/arrow/pull/6323#issuecomment-580137629.

When the size of the ArrowBuf exceeds 2GB, our flighing library does not work due to integer overflow. 

This is because internally, we have used some data structures which are based on 32-bit integers. To resolve the problem, we must revise/replace the data structures to make them support 64-bit integers. 

As a concrete example, we can see that when the server sends data through IPC, an org.apache.arrow.flight.ArrowMessage object is created, and is wrapped as an InputStream through the `asInputStream` method. In this method, we use data stuctures like java.io.ByteArrayOutputStream and io.netty.buffer.ByteBuf, which are based on 32-bit integers (we can observe that NettyArrowBuf#length and ByteArrayOutputStream#count are both 32-bit integers). 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)