You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Jenny Kwan <me...@jennykwan.org> on 2019/06/21 02:03:04 UTC

JS - proper way to instantiate a RecordBatch with nulls

Apologies if this has been discussed before. I'm attempting to create Arrow files in an Electron desktop app using JS. I can't find docs showing how to create a RecordBatch with a column of, say, nullable bools. Using Table.from and Table.new both keep coercing nulls to falses. Any pointers? 

Thanks, 
Jenny

⁣Get BlueMail for Android

Re: JS - proper way to instantiate a RecordBatch with nulls

Posted by Jenny Kwan <me...@jennykwan.org>.

Thanks, Paul! 

I'll use pre-release. For my own edification I'll take a look at builder implementation. 

Thanks! 
Jenny

⁣Get BlueMail for Android 

On Jun 21, 2019, 02:51, at 02:51, Paul Taylor <pt...@gmail.com> wrote:
>Hi Jenny,
>
>In the 0.13 you have to pre-allocate the null bitmap and data buffers 
>ahead of time, then use `vec.set(idx, val)` to write each value in.
>
>In 0.14 you can instead use the new Builders from PR #4476 
><https://github.com/apache/arrow/pull/4476>. These will create and 
>resize the underlying buffers as you append values, and includes 
>chunking and streaming APIs for convenience. You can use them by 
>pointing npm to my github repo like I did here 
><https://github.com/trxcllnt/csv-to-arrow-js/blob/2c625fb4b47a84044d62710f63c139a054281d83/package.json#L12>
>
>if you don't mind using a prerelease build until 0.14 is out.
>
>Best,
>Paul
>
>On 6/20/19 7:03 PM, Jenny Kwan wrote:
>> Apologies if this has been discussed before. I'm attempting to create
>Arrow files in an Electron desktop app using JS. I can't find docs
>showing how to create a RecordBatch with a column of, say, nullable
>bools. Using Table.from and Table.new both keep coercing nulls to
>falses. Any pointers?
>>
>> Thanks,
>> Jenny
>>
>> ⁣Get BlueMail for Android

Re: JS - proper way to instantiate a RecordBatch with nulls

Posted by Paul Taylor <pt...@gmail.com>.

Hi Jenny,

In the 0.13 you have to pre-allocate the null bitmap and data buffers 
ahead of time, then use `vec.set(idx, val)` to write each value in.

In 0.14 you can instead use the new Builders from PR #4476 
<https://github.com/apache/arrow/pull/4476>. These will create and 
resize the underlying buffers as you append values, and includes 
chunking and streaming APIs for convenience. You can use them by 
pointing npm to my github repo like I did here 
<https://github.com/trxcllnt/csv-to-arrow-js/blob/2c625fb4b47a84044d62710f63c139a054281d83/package.json#L12> 
if you don't mind using a prerelease build until 0.14 is out.

Best,
Paul

On 6/20/19 7:03 PM, Jenny Kwan wrote:
> Apologies if this has been discussed before. I'm attempting to create Arrow files in an Electron desktop app using JS. I can't find docs showing how to create a RecordBatch with a column of, say, nullable bools. Using Table.from and Table.new both keep coercing nulls to falses. Any pointers?
>
> Thanks,
> Jenny
>
> ⁣Get BlueMail for Android