You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Tanveer Ahmad - EWI <T....@tudelft.nl> on 2019/03/24 12:33:01 UTC
Java OutOfMemoryException!
Hi,
I am de-serializing multiple plasma objects in java at the same time, everything is working fine but when the data size increases the following error is being occurred for some threads. Any suggestion where I can increase/change the memory allocation for these processes (I have more memory available)? Is it JVM related or Arrow specific?
Exception in thread "Thread-1" org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 634729984 due to memory limit. Current allocation: 0
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273)
at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249)
at org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88)
at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204)
at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116)
Thanks.
Regards,
Tanveer Ahmad
Re: Java OutOfMemoryException!
Posted by Tanveer Ahmad - EWI <T....@tudelft.nl>.
Thanks Razvan.
Increasing the RootAllocator limit resolved the problem.
Regards,
Tanveer Ahmad
________________________________
From: Razvan Chitu <ra...@gmail.com>
Sent: Sunday, March 24, 2019 2:44:57 PM
To: dev@arrow.apache.org
Cc: user@arrow.apache.org
Subject: Re: Java OutOfMemoryException!
Hi Tanveer,
The stack trace seems to indicate that you you've breached the limit of the
allocator used by the ArrowStreamReader, so that's where I'd look first.
The limit is usually set when constructing an allocator (e.g. new
RootAllocator(myLimit)) or when getting a child allocator (e.g.
rootAllocator.newChildAllocator(...)).
Razvan
On Sun, Mar 24, 2019 at 12:33 PM Tanveer Ahmad - EWI <T....@tudelft.nl>
wrote:
> Hi,
>
> I am de-serializing multiple plasma objects in java at the same time,
> everything is working fine but when the data size increases the following
> error is being occurred for some threads. Any suggestion where I can
> increase/change the memory allocation for these processes (I have more
> memory available)? Is it JVM related or Arrow specific?
>
> Exception in thread "Thread-1"
> org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of
> size 634729984 due to memory limit. Current allocation: 0
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273)
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249)
> at
> org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204)
> at
> org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116)
>
>
>
> Thanks.
>
>
> Regards,
> Tanveer Ahmad
>
Re: Java OutOfMemoryException!
Posted by Tanveer Ahmad - EWI <T....@tudelft.nl>.
Thanks Razvan.
Increasing the RootAllocator limit resolved the problem.
Regards,
Tanveer Ahmad
________________________________
From: Razvan Chitu <ra...@gmail.com>
Sent: Sunday, March 24, 2019 2:44:57 PM
To: dev@arrow.apache.org
Cc: user@arrow.apache.org
Subject: Re: Java OutOfMemoryException!
Hi Tanveer,
The stack trace seems to indicate that you you've breached the limit of the
allocator used by the ArrowStreamReader, so that's where I'd look first.
The limit is usually set when constructing an allocator (e.g. new
RootAllocator(myLimit)) or when getting a child allocator (e.g.
rootAllocator.newChildAllocator(...)).
Razvan
On Sun, Mar 24, 2019 at 12:33 PM Tanveer Ahmad - EWI <T....@tudelft.nl>
wrote:
> Hi,
>
> I am de-serializing multiple plasma objects in java at the same time,
> everything is working fine but when the data size increases the following
> error is being occurred for some threads. Any suggestion where I can
> increase/change the memory allocation for these processes (I have more
> memory available)? Is it JVM related or Arrow specific?
>
> Exception in thread "Thread-1"
> org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of
> size 634729984 due to memory limit. Current allocation: 0
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273)
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249)
> at
> org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204)
> at
> org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116)
>
>
>
> Thanks.
>
>
> Regards,
> Tanveer Ahmad
>
Re: Java OutOfMemoryException!
Posted by Razvan Chitu <ra...@gmail.com>.
Hi Tanveer,
The stack trace seems to indicate that you you've breached the limit of the
allocator used by the ArrowStreamReader, so that's where I'd look first.
The limit is usually set when constructing an allocator (e.g. new
RootAllocator(myLimit)) or when getting a child allocator (e.g.
rootAllocator.newChildAllocator(...)).
Razvan
On Sun, Mar 24, 2019 at 12:33 PM Tanveer Ahmad - EWI <T....@tudelft.nl>
wrote:
> Hi,
>
> I am de-serializing multiple plasma objects in java at the same time,
> everything is working fine but when the data size increases the following
> error is being occurred for some threads. Any suggestion where I can
> increase/change the memory allocation for these processes (I have more
> memory available)? Is it JVM related or Arrow specific?
>
> Exception in thread "Thread-1"
> org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of
> size 634729984 due to memory limit. Current allocation: 0
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273)
> at
> org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249)
> at
> org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88)
> at
> org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204)
> at
> org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116)
>
>
>
> Thanks.
>
>
> Regards,
> Tanveer Ahmad
>