[jira] [Created] (ARROW-5386) Making the rounding behavior of the buffer capacity configurable

["Liya Fan (JIRA)" <ji...@apache.org>]

Liya Fan created ARROW-5386:
-------------------------------

             Summary: Making the rounding behavior of the buffer capacity configurable
                 Key: ARROW-5386
                 URL: https://issues.apache.org/jira/browse/ARROW-5386
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Java
            Reporter: Liya Fan
            Assignee: Liya Fan


In our scenario, the following code snippet is frequent in our code base:

int requestSize = ...;
if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory()) {
 ArrowBuf buffer = allocator.buffer(requestSize);
}

However, it often causes OutOfMemoryException, due to Arrow's rounding behavior.

For example, we have only 12 MB memory left, and we request a buffer with size 10 MB. Appearantly, there is sufficient memory to meet the request. However, the rounding behavior rounds the request size from 10 MB to 16 MB, and there is no 16 MB memory, so an OutOfMemoryException will be thrown.

We propose two ways to solve this problem:

1. We provide a rounding option as an argument to the BaseAllocator#buf method. There are two possible values for the rounding option: rounding up and rounding down. In the above scenario, the rounding down option can solve the problem.

2. We add a method to the allocator:

int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption roundingOption)

This method will give the rounding buffer size, given the initial request size. With this method, the user can freely adjust their request size to avoid OOM.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)