[jira] [Created] (ARROW-1533) As part of buffer transfer (transferTo function), we should transfer the state for realloc

["Siddharth Teotia (JIRA)" <ji...@apache.org>]

Siddharth Teotia created ARROW-1533:
---------------------------------------

             Summary: As part of buffer transfer (transferTo function), we should transfer the state for realloc
                 Key: ARROW-1533
                 URL: https://issues.apache.org/jira/browse/ARROW-1533
             Project: Apache Arrow
          Issue Type: Bug
          Components: Java - Vectors
            Reporter: Siddharth Teotia
            Assignee: Siddharth Teotia


We recently encountered a problem when we were trying to add JSON files with complex schema as datasets.

Initially we started with a Float8Vector with default memory allocation of (4096 * 8) 32KB.
Went through several iterations of setSafe() to trigger a realloc() from 32KB to 64KB.
Another round of setSafe() calls to trigger a realloc() from 64KB to 128KB

After that we encountered a BigInt and promoted our vector to UnionVector.

This required us to create a UnionVector with BigIntVector and Float8Vector. The latter required us to transfer the Float8Vector we were earlier working with to the Float8Vector inside the Union.

As part of transferTo(), the target Float8Vector got all the ArrowBuf state (capacity, buffer contents) etc transferred from the source vector.

Later, a realloc was triggered on the Float8Vector inside the UnionVector.

The computation inside realloc() to determine the amount of memory to be reallocated goes wrong since it makes the decision based on allocateSizeInBytes -- although this vector was created as part of transfer() from 128KB source vector, allocateSizeInBytes is still at the initial/default value of 32KB

We end up allocating a 64KB buffer and attempt to copy 128KB over 64KB and seg fault when invoking setBytes().

There is a wrong assumption in realloc() that allocateSizeInBytes is always equal to data.capacity(). The particular scenario described above exposes where this assumption could go wrong.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)