Bulk copy methods to/from Java vectors

[Chris Nuernberger <ch...@techascent.com>]

Hey,

I would like to have bulk methods for copying data into a vector.
Specifically, I have an existing data table so I know the desired lengths
of the columns.  I can also precalculate the necessary buffer sizes for any
variable sized column.


What I don't see is how to pre-allocate columns of a given size.  When I
use setValueCount on a column and then use the set method I get a netty
error.  What I was hoping for is some allocation method, especially for
primitive data, that allocates the desired uninitialized memory for the
valide and buffer data and then hands those two buffers back to me so I can
use memcpy and friends as opposed to repeated calls to setSafe.


Repeated calls to setSafe are time consuming, not parallelizable, and
unnecessary when I know the data rectangle I would like to transfer into a
record batch.


In my case I have the data pre-cut.  How would you recommend copying bulk
portions of data (that may be in java arrays or in some more abstract
interface) into a record batch?

Thanks for any help,

Chris

Re: Bulk copy methods to/from Java vectors

[Chris Nuernberger <ch...@techascent.com>]

Make sense also, thank you for your help.  Calling setValueCount after
allocateNew solved the problem apparently at least allowing me to
round-trip some data.  I never called setLastSet once so perhaps there was
some duplicate work done.

On Sun, Jul 26, 2020 at 10:30 AM Jacques Nadeau <ja...@apache.org> wrote:

>
> On Sun, Jul 26, 2020 at 8:02 AM Chris Nuernberger <ch...@techascent.com>
> wrote:
>
>> It appears that those methods do not allocate the validity buffer *and*
>> the function `allocateValidityBuffer` is private.
>>
>
> It allocates both of them at once. To reduce heap usage we colocate them
> since they are never resized indepently.
>
>
> Also it appears that allocate new fails to set the value count for
>> BaseVariableWidthVectors.  And if you set the value count after you have
>> assigned data then it clears *only* the offset buffer but not the validity
>> or the data buffers.
>
>
> For direct operations on variable, you'll need to do the following steps:
> 1) allocateNew,
> 2) copy in data via memory operations,
> 3) call setLastSet()
> 4) call setValueCount()
>
> I'm guessing you skipped #3 and then setValueCount sees that you never set
> any values so it propagates the the last offset to the value count. This is
> done so you can do something like:
> set(1,...)
> set(3,...)
> setValueCount(7)
> and then 4-6 ordinal positions will be offset filled even though you
> didn't set them explicitly. If you do your own work, you have to help the
> state model in the variable vector understand what you've done.
>

Re: Bulk copy methods to/from Java vectors

[Jacques Nadeau <ja...@apache.org>]

On Sun, Jul 26, 2020 at 8:02 AM Chris Nuernberger <ch...@techascent.com>
wrote:

> It appears that those methods do not allocate the validity buffer *and*
> the function `allocateValidityBuffer` is private.
>

It allocates both of them at once. To reduce heap usage we colocate them
since they are never resized indepently.


Also it appears that allocate new fails to set the value count for
> BaseVariableWidthVectors.  And if you set the value count after you have
> assigned data then it clears *only* the offset buffer but not the validity
> or the data buffers.


For direct operations on variable, you'll need to do the following steps:
1) allocateNew,
2) copy in data via memory operations,
3) call setLastSet()
4) call setValueCount()

I'm guessing you skipped #3 and then setValueCount sees that you never set
any values so it propagates the the last offset to the value count. This is
done so you can do something like:
set(1,...)
set(3,...)
setValueCount(7)
and then 4-6 ordinal positions will be offset filled even though you didn't
set them explicitly. If you do your own work, you have to help the state
model in the variable vector understand what you've done.

Re: Bulk copy methods to/from Java vectors

[Chris Nuernberger <ch...@techascent.com>]

Also it appears that allocate new fails to set the value count for
BaseVariableWidthVectors.  And if you set the value count after you have
assigned data then it clears *only* the offset buffer but not the validity
or the data buffers.

On Sun, Jul 26, 2020 at 9:01 AM Chris Nuernberger <ch...@techascent.com>
wrote:

> It appears that those methods do not allocate the validity buffer *and*
> the function `allocateValidityBuffer` is private.
>
> How do you recommend allocating the validity buffer?
>
> On Sun, Jul 26, 2020 at 6:48 AM Chris Nuernberger <ch...@techascent.com>
> wrote:
>
>> Perfect, thank you.  I tried setCapacity and setValueCount together and
>> this didn't have the result I was hoping for.  The methods you provide are
>> what I was looking for.
>>
>> On Sat, Jul 25, 2020 at 5:22 PM Jacques Nadeau <ja...@apache.org>
>> wrote:
>>
>>> You can allocate exactly for both fixed [1] and variable types [2].
>>>
>>> 1:
>>> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java#L292
>>> 2:
>>> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L401
>>>
>>> You can then use the set method per cell or just grab the memory address
>>> (e.g. getDataBufferAddress()) and use Unsafe to bulk copy. The latter
>>> obviously is more advanced and requires you do things like set the
>>> validity buffers as well.
>>>
>>>
>>> On Sat, Jul 25, 2020 at 6:02 AM Chris Nuernberger <ch...@techascent.com>
>>> wrote:
>>>
>>>> Hey,
>>>>
>>>> I would like to have bulk methods for copying data into a vector.
>>>> Specifically, I have an existing data table so I know the desired lengths
>>>> of the columns.  I can also precalculate the necessary buffer sizes for any
>>>> variable sized column.
>>>>
>>>>
>>>> What I don't see is how to pre-allocate columns of a given size.  When
>>>> I use setValueCount on a column and then use the set method I get a netty
>>>> error.  What I was hoping for is some allocation method, especially for
>>>> primitive data, that allocates the desired uninitialized memory for the
>>>> valide and buffer data and then hands those two buffers back to me so I can
>>>> use memcpy and friends as opposed to repeated calls to setSafe.
>>>>
>>>>
>>>> Repeated calls to setSafe are time consuming, not parallelizable, and
>>>> unnecessary when I know the data rectangle I would like to transfer into a
>>>> record batch.
>>>>
>>>>
>>>> In my case I have the data pre-cut.  How would you recommend copying
>>>> bulk portions of data (that may be in java arrays or in some more abstract
>>>> interface) into a record batch?
>>>>
>>>> Thanks for any help,
>>>>
>>>> Chris
>>>>
>>>

Re: Bulk copy methods to/from Java vectors

[Chris Nuernberger <ch...@techascent.com>]

It appears that those methods do not allocate the validity buffer *and* the
function `allocateValidityBuffer` is private.

How do you recommend allocating the validity buffer?

On Sun, Jul 26, 2020 at 6:48 AM Chris Nuernberger <ch...@techascent.com>
wrote:

> Perfect, thank you.  I tried setCapacity and setValueCount together and
> this didn't have the result I was hoping for.  The methods you provide are
> what I was looking for.
>
> On Sat, Jul 25, 2020 at 5:22 PM Jacques Nadeau <ja...@apache.org> wrote:
>
>> You can allocate exactly for both fixed [1] and variable types [2].
>>
>> 1:
>> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java#L292
>> 2:
>> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L401
>>
>> You can then use the set method per cell or just grab the memory address
>> (e.g. getDataBufferAddress()) and use Unsafe to bulk copy. The latter
>> obviously is more advanced and requires you do things like set the
>> validity buffers as well.
>>
>>
>> On Sat, Jul 25, 2020 at 6:02 AM Chris Nuernberger <ch...@techascent.com>
>> wrote:
>>
>>> Hey,
>>>
>>> I would like to have bulk methods for copying data into a vector.
>>> Specifically, I have an existing data table so I know the desired lengths
>>> of the columns.  I can also precalculate the necessary buffer sizes for any
>>> variable sized column.
>>>
>>>
>>> What I don't see is how to pre-allocate columns of a given size.  When I
>>> use setValueCount on a column and then use the set method I get a netty
>>> error.  What I was hoping for is some allocation method, especially for
>>> primitive data, that allocates the desired uninitialized memory for the
>>> valide and buffer data and then hands those two buffers back to me so I can
>>> use memcpy and friends as opposed to repeated calls to setSafe.
>>>
>>>
>>> Repeated calls to setSafe are time consuming, not parallelizable, and
>>> unnecessary when I know the data rectangle I would like to transfer into a
>>> record batch.
>>>
>>>
>>> In my case I have the data pre-cut.  How would you recommend copying
>>> bulk portions of data (that may be in java arrays or in some more abstract
>>> interface) into a record batch?
>>>
>>> Thanks for any help,
>>>
>>> Chris
>>>
>>

Re: Bulk copy methods to/from Java vectors

[Chris Nuernberger <ch...@techascent.com>]

Perfect, thank you.  I tried setCapacity and setValueCount together and
this didn't have the result I was hoping for.  The methods you provide are
what I was looking for.

On Sat, Jul 25, 2020 at 5:22 PM Jacques Nadeau <ja...@apache.org> wrote:

> You can allocate exactly for both fixed [1] and variable types [2].
>
> 1:
> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java#L292
> 2:
> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L401
>
> You can then use the set method per cell or just grab the memory address
> (e.g. getDataBufferAddress()) and use Unsafe to bulk copy. The latter
> obviously is more advanced and requires you do things like set the
> validity buffers as well.
>
>
> On Sat, Jul 25, 2020 at 6:02 AM Chris Nuernberger <ch...@techascent.com>
> wrote:
>
>> Hey,
>>
>> I would like to have bulk methods for copying data into a vector.
>> Specifically, I have an existing data table so I know the desired lengths
>> of the columns.  I can also precalculate the necessary buffer sizes for any
>> variable sized column.
>>
>>
>> What I don't see is how to pre-allocate columns of a given size.  When I
>> use setValueCount on a column and then use the set method I get a netty
>> error.  What I was hoping for is some allocation method, especially for
>> primitive data, that allocates the desired uninitialized memory for the
>> valide and buffer data and then hands those two buffers back to me so I can
>> use memcpy and friends as opposed to repeated calls to setSafe.
>>
>>
>> Repeated calls to setSafe are time consuming, not parallelizable, and
>> unnecessary when I know the data rectangle I would like to transfer into a
>> record batch.
>>
>>
>> In my case I have the data pre-cut.  How would you recommend copying bulk
>> portions of data (that may be in java arrays or in some more abstract
>> interface) into a record batch?
>>
>> Thanks for any help,
>>
>> Chris
>>
>

Re: Bulk copy methods to/from Java vectors

[Jacques Nadeau <ja...@apache.org>]

You can allocate exactly for both fixed [1] and variable types [2].

1:
https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java#L292
2:
https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java#L401

You can then use the set method per cell or just grab the memory address
(e.g. getDataBufferAddress()) and use Unsafe to bulk copy. The latter
obviously is more advanced and requires you do things like set the
validity buffers as well.


On Sat, Jul 25, 2020 at 6:02 AM Chris Nuernberger <ch...@techascent.com>
wrote:

> Hey,
>
> I would like to have bulk methods for copying data into a vector.
> Specifically, I have an existing data table so I know the desired lengths
> of the columns.  I can also precalculate the necessary buffer sizes for any
> variable sized column.
>
>
> What I don't see is how to pre-allocate columns of a given size.  When I
> use setValueCount on a column and then use the set method I get a netty
> error.  What I was hoping for is some allocation method, especially for
> primitive data, that allocates the desired uninitialized memory for the
> valide and buffer data and then hands those two buffers back to me so I can
> use memcpy and friends as opposed to repeated calls to setSafe.
>
>
> Repeated calls to setSafe are time consuming, not parallelizable, and
> unnecessary when I know the data rectangle I would like to transfer into a
> record batch.
>
>
> In my case I have the data pre-cut.  How would you recommend copying bulk
> portions of data (that may be in java arrays or in some more abstract
> interface) into a record batch?
>
> Thanks for any help,
>
> Chris
>