[jira] [Closed] (ARROW-5386) Making the rounding behavior of the buffer capacity configurable

["Liya Fan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Liya Fan closed ARROW-5386.
---------------------------
    Resolution: Workaround

ARROW-5429 provides a workaround for this issue. 

That is, the user can get to know the actual buffer size through the RoundingStrategy object. So OOM can be avoided by adjusting the request buffer size. 

> Making the rounding behavior of the buffer capacity configurable
> ----------------------------------------------------------------
>
>                 Key: ARROW-5386
>                 URL: https://issues.apache.org/jira/browse/ARROW-5386
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Java
>            Reporter: Liya Fan
>            Assignee: Liya Fan
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> In our scenario, the following code snippet is frequent in our code base:
> int requestSize = ...;
>  if (requestSize <= allocator.getLimit() - allocator.getAllocatedMemory())
> { ArrowBuf buffer = allocator.buffer(requestSize); }
> However, it often causes OutOfMemoryException, due to Arrow's rounding behavior.
> For example, we have only 12 MB memory left, and we request a buffer with size 10 MB. Appearantly, there is sufficient memory to meet the request. However, the rounding behavior rounds the request size from 10 MB to 16 MB, and there is no 16 MB memory, so an OutOfMemoryException will be thrown.
> We propose two ways to solve this problem:
> 1. We provide a rounding option as an argument to the BaseAllocator#buffer method. There are two possible values for the rounding option: rounding up and rounding down. In the above scenario, the rounding down option can solve the problem.
> 2. We add a method to the allocator:
> int getRoundedSize(final int size, BaseAllocator.AllocationRoundingOption roundingOption)
> This method will give the rounded buffer size, given the initial request size. With this method, the user can freely adjust their request size to avoid OOM.
> To make it more convenient to use Arrow, we think both solutions should be implemented.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)