You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Felipe Aramburu <fe...@blazingdb.com> on 2019/03/22 14:40:56 UTC
Creating Arrays from builders using bitmasks
In the builder base class I see this api
https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
// Vector append. Treat each zero byte as a nullzero. If valid_bytes is
null
// assume all of length bits are valid.
void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t length)
Is valid_bytes an allocation of size (int8_t) * length, using an entire
byte to indicate validity for each element in the array or is this a null
bitmask where in each byte in valid_bytes encodes 8 values, one per bit?
If this is using a byte per value is there an approved way of using a
builder to initialize an array using the memory layout described here
https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
Re: Creating Arrays from builders using bitmasks
Posted by Wes McKinney <we...@gmail.com>.
hi Felipe -- note you don't have to use the builder classes when you
have the exact memory layout already, you can wrap you rmemory in
arrow::Buffer and construct the arrays directly.
I think it would be useful to add APIs for appending to builders with
a bitmap. We don't have them now, though. This would be a useful
contribution to the project.
- Wes
On Fri, Mar 22, 2019 at 10:26 AM Felipe Aramburu <fe...@blazingdb.com> wrote:
>
> Is there a way to use a builder to be able to provide a bit per value LSB
> as is described in the documentation? I have this already and it seems
> silly to convert it to something else so that arrow can then make it the
> same format as what I had to begin with. I know there is the ArrayData
> class that has a make function that seems to allow me to do this but I am
> trying to do things as the documentation suggest which I assumed was the
> preferred method of doing this.
>
>
>
> On Fri, Mar 22, 2019 at 8:13 AM Francois Saint-Jacques <
> fsaintjacques@gmail.com> wrote:
>
> > Actually, this specific method seems to use a byte per value as you
> > questioned. I think it's worth adding documentation and an explicit warning
> > if it confused me. I'll let bkietz chime in to comment on the usage.
> >
> > François
> >
> > On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques <
> > fsaintjacques@gmail.com> wrote:
> >
> > > Hello Felipe,
> > >
> > > it's a bit per value as per memory layout documentation.
> > >
> > > François
> > >
> > >
> > >
> > > On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <fe...@blazingdb.com>
> > > wrote:
> > >
> > >> In the builder base class I see this api
> > >>
> > >>
> > >>
> > https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
> > >>
> > >> // Vector append. Treat each zero byte as a nullzero. If valid_bytes
> > >> is
> > >> null
> > >> // assume all of length bits are valid.
> > >> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t
> > length)
> > >>
> > >> Is valid_bytes an allocation of size (int8_t) * length, using an entire
> > >> byte to indicate validity for each element in the array or is this a
> > null
> > >> bitmask where in each byte in valid_bytes encodes 8 values, one per bit?
> > >>
> > >> If this is using a byte per value is there an approved way of using a
> > >> builder to initialize an array using the memory layout described here
> > >> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
> > >>
> > >
> >
Re: Creating Arrays from builders using bitmasks
Posted by Felipe Aramburu <fe...@blazingdb.com>.
Got it. I think I am just going to use arraydata::make for this for now.
Thanks a bundle!
Felipe
On Fri, Mar 22, 2019 at 8:43 AM Francois Saint-Jacques <
fsaintjacques@gmail.com> wrote:
> It is not frowned upon to use the ArrayData::Make classes, you just have to
> ensure the order of buffers matches what the specialized Array class
> expects (also matching the type you're passing to ArrayData). I'd say it is
> the "preferred" way if your data is already in the required layout, and the
> best way if you want effective zero-copy.
>
> I created https://issues.apache.org/jira/browse/ARROW-4999 to track
> improving the documentation.
>
> François
>
> On Fri, Mar 22, 2019 at 11:26 AM Felipe Aramburu <fe...@blazingdb.com>
> wrote:
>
> > Is there a way to use a builder to be able to provide a bit per value LSB
> > as is described in the documentation? I have this already and it seems
> > silly to convert it to something else so that arrow can then make it the
> > same format as what I had to begin with. I know there is the ArrayData
> > class that has a make function that seems to allow me to do this but I am
> > trying to do things as the documentation suggest which I assumed was the
> > preferred method of doing this.
> >
> >
> >
> > On Fri, Mar 22, 2019 at 8:13 AM Francois Saint-Jacques <
> > fsaintjacques@gmail.com> wrote:
> >
> > > Actually, this specific method seems to use a byte per value as you
> > > questioned. I think it's worth adding documentation and an explicit
> > warning
> > > if it confused me. I'll let bkietz chime in to comment on the usage.
> > >
> > > François
> > >
> > > On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques <
> > > fsaintjacques@gmail.com> wrote:
> > >
> > > > Hello Felipe,
> > > >
> > > > it's a bit per value as per memory layout documentation.
> > > >
> > > > François
> > > >
> > > >
> > > >
> > > > On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <
> felipe@blazingdb.com
> > >
> > > > wrote:
> > > >
> > > >> In the builder base class I see this api
> > > >>
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
> > > >>
> > > >> // Vector append. Treat each zero byte as a nullzero. If
> > valid_bytes
> > > >> is
> > > >> null
> > > >> // assume all of length bits are valid.
> > > >> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t
> > > length)
> > > >>
> > > >> Is valid_bytes an allocation of size (int8_t) * length, using an
> > entire
> > > >> byte to indicate validity for each element in the array or is this a
> > > null
> > > >> bitmask where in each byte in valid_bytes encodes 8 values, one per
> > bit?
> > > >>
> > > >> If this is using a byte per value is there an approved way of using
> a
> > > >> builder to initialize an array using the memory layout described
> here
> > > >> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
> > > >>
> > > >
> > >
> >
>
Re: Creating Arrays from builders using bitmasks
Posted by Francois Saint-Jacques <fs...@gmail.com>.
It is not frowned upon to use the ArrayData::Make classes, you just have to
ensure the order of buffers matches what the specialized Array class
expects (also matching the type you're passing to ArrayData). I'd say it is
the "preferred" way if your data is already in the required layout, and the
best way if you want effective zero-copy.
I created https://issues.apache.org/jira/browse/ARROW-4999 to track
improving the documentation.
François
On Fri, Mar 22, 2019 at 11:26 AM Felipe Aramburu <fe...@blazingdb.com>
wrote:
> Is there a way to use a builder to be able to provide a bit per value LSB
> as is described in the documentation? I have this already and it seems
> silly to convert it to something else so that arrow can then make it the
> same format as what I had to begin with. I know there is the ArrayData
> class that has a make function that seems to allow me to do this but I am
> trying to do things as the documentation suggest which I assumed was the
> preferred method of doing this.
>
>
>
> On Fri, Mar 22, 2019 at 8:13 AM Francois Saint-Jacques <
> fsaintjacques@gmail.com> wrote:
>
> > Actually, this specific method seems to use a byte per value as you
> > questioned. I think it's worth adding documentation and an explicit
> warning
> > if it confused me. I'll let bkietz chime in to comment on the usage.
> >
> > François
> >
> > On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques <
> > fsaintjacques@gmail.com> wrote:
> >
> > > Hello Felipe,
> > >
> > > it's a bit per value as per memory layout documentation.
> > >
> > > François
> > >
> > >
> > >
> > > On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <felipe@blazingdb.com
> >
> > > wrote:
> > >
> > >> In the builder base class I see this api
> > >>
> > >>
> > >>
> >
> https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
> > >>
> > >> // Vector append. Treat each zero byte as a nullzero. If
> valid_bytes
> > >> is
> > >> null
> > >> // assume all of length bits are valid.
> > >> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t
> > length)
> > >>
> > >> Is valid_bytes an allocation of size (int8_t) * length, using an
> entire
> > >> byte to indicate validity for each element in the array or is this a
> > null
> > >> bitmask where in each byte in valid_bytes encodes 8 values, one per
> bit?
> > >>
> > >> If this is using a byte per value is there an approved way of using a
> > >> builder to initialize an array using the memory layout described here
> > >> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
> > >>
> > >
> >
>
Re: Creating Arrays from builders using bitmasks
Posted by Felipe Aramburu <fe...@blazingdb.com>.
Is there a way to use a builder to be able to provide a bit per value LSB
as is described in the documentation? I have this already and it seems
silly to convert it to something else so that arrow can then make it the
same format as what I had to begin with. I know there is the ArrayData
class that has a make function that seems to allow me to do this but I am
trying to do things as the documentation suggest which I assumed was the
preferred method of doing this.
On Fri, Mar 22, 2019 at 8:13 AM Francois Saint-Jacques <
fsaintjacques@gmail.com> wrote:
> Actually, this specific method seems to use a byte per value as you
> questioned. I think it's worth adding documentation and an explicit warning
> if it confused me. I'll let bkietz chime in to comment on the usage.
>
> François
>
> On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques <
> fsaintjacques@gmail.com> wrote:
>
> > Hello Felipe,
> >
> > it's a bit per value as per memory layout documentation.
> >
> > François
> >
> >
> >
> > On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <fe...@blazingdb.com>
> > wrote:
> >
> >> In the builder base class I see this api
> >>
> >>
> >>
> https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
> >>
> >> // Vector append. Treat each zero byte as a nullzero. If valid_bytes
> >> is
> >> null
> >> // assume all of length bits are valid.
> >> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t
> length)
> >>
> >> Is valid_bytes an allocation of size (int8_t) * length, using an entire
> >> byte to indicate validity for each element in the array or is this a
> null
> >> bitmask where in each byte in valid_bytes encodes 8 values, one per bit?
> >>
> >> If this is using a byte per value is there an approved way of using a
> >> builder to initialize an array using the memory layout described here
> >> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
> >>
> >
>
Re: Creating Arrays from builders using bitmasks
Posted by Francois Saint-Jacques <fs...@gmail.com>.
Actually, this specific method seems to use a byte per value as you
questioned. I think it's worth adding documentation and an explicit warning
if it confused me. I'll let bkietz chime in to comment on the usage.
François
On Fri, Mar 22, 2019 at 10:57 AM Francois Saint-Jacques <
fsaintjacques@gmail.com> wrote:
> Hello Felipe,
>
> it's a bit per value as per memory layout documentation.
>
> François
>
>
>
> On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <fe...@blazingdb.com>
> wrote:
>
>> In the builder base class I see this api
>>
>>
>> https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
>>
>> // Vector append. Treat each zero byte as a nullzero. If valid_bytes
>> is
>> null
>> // assume all of length bits are valid.
>> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t length)
>>
>> Is valid_bytes an allocation of size (int8_t) * length, using an entire
>> byte to indicate validity for each element in the array or is this a null
>> bitmask where in each byte in valid_bytes encodes 8 values, one per bit?
>>
>> If this is using a byte per value is there an approved way of using a
>> builder to initialize an array using the memory layout described here
>> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
>>
>
Re: Creating Arrays from builders using bitmasks
Posted by Francois Saint-Jacques <fs...@gmail.com>.
Hello Felipe,
it's a bit per value as per memory layout documentation.
François
On Fri, Mar 22, 2019 at 10:48 AM Felipe Aramburu <fe...@blazingdb.com>
wrote:
> In the builder base class I see this api
>
>
> https://github.com/apache/arrow/blob/ad1697e5d25eeaff5630421f55b0120f45cf0ce1/cpp/src/arrow/array/builder_base.h#L149
>
> // Vector append. Treat each zero byte as a nullzero. If valid_bytes is
> null
> // assume all of length bits are valid.
> void UnsafeAppendToBitmap(const uint8_t* valid_bytes, int64_t length)
>
> Is valid_bytes an allocation of size (int8_t) * length, using an entire
> byte to indicate validity for each element in the array or is this a null
> bitmask where in each byte in valid_bytes encodes 8 values, one per bit?
>
> If this is using a byte per value is there an approved way of using a
> builder to initialize an array using the memory layout described here
> https://arrow.apache.org/docs/memory_layout.html#null-bitmaps?
>