[jira] [Updated] (ARROW-1533) [JAVA] realloc should consider the existing buffer capacity for computing target memory requirement

["Siddharth Teotia (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Teotia updated ARROW-1533:
------------------------------------
    Summary: [JAVA] realloc should consider the existing buffer capacity for computing target memory requirement  (was: As part of buffer transfer (transferTo function), we should transfer the state for realloc)

> [JAVA] realloc should consider the existing buffer capacity for computing target memory requirement
> ---------------------------------------------------------------------------------------------------
>
>                 Key: ARROW-1533
>                 URL: https://issues.apache.org/jira/browse/ARROW-1533
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Java - Vectors
>            Reporter: Siddharth Teotia
>            Assignee: Siddharth Teotia
>
> We recently encountered a problem when we were trying to add JSON files with complex schema as datasets.
> Initially we started with a Float8Vector with default memory allocation of (4096 * 8) 32KB.
> Went through several iterations of setSafe() to trigger a realloc() from 32KB to 64KB.
> Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB
> After that we encountered a BigInt and promoted our vector to UnionVector.
> This required us to create a UnionVector with BigIntVector and Float8Vector. The latter required us to transfer the Float8Vector we were earlier working with to the Float8Vector inside the Union.
> As part of transferTo(), the target Float8Vector got all the ArrowBuf state (capacity, buffer contents) etc transferred from the source vector.
> Later, a realloc was triggered on the Float8Vector inside the UnionVector.
> The computation inside realloc() to determine the amount of memory to be reallocated goes wrong since it makes the decision based on allocateSizeInBytes -- although this vector was created as part of transfer() from 128KB source vector, allocateSizeInBytes is still at the initial/default value of 32KB
> We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and seg fault when invoking setBytes().
> There is a wrong assumption in realloc() that allocateSizeInBytes is always equal to data.capacity(). The particular scenario described above exposes where this assumption could go wrong.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)