You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Micah Kornfield <em...@gmail.com> on 2019/03/27 06:51:38 UTC
[Format] For all null arrays are data-buffers required?
Similar to how there is no validity-buffer required in the format if
null_count == 0, is a similar optimization for the "data buffer" allowed
when (null_count == array length)? It seems that if all values are null,
no data element should ever be accessed, but I couldn't find if this was
ever discussed.
A quick perusal of the spec seems to imply they are required (I couldn't
find an exception). But I just wanted to confirm this was the case.
Thanks,
Micah
Re: [Format] For all null arrays are data-buffers required?
Posted by Wes McKinney <we...@gmail.com>.
Let me fix a typo
On Wed, Mar 27, 2019 at 7:40 AM Wes McKinney <we...@gmail.com> wrote:
>
> hi Micah,
>
> I think it's most productive to view things through the lens of the
> binary protocol (i.e. the IPC/RPC wire format)
>
> On Wed, Mar 27, 2019 at 1:59 AM Micah Kornfield <em...@gmail.com> wrote:
> >
> > Similar to how there is no validity-buffer required in the format if
> > null_count == 0, is a similar optimization for the "data buffer" allowed
> > when (null_count == array length)? It seems that if all values are null,
> > no data element should ever be accessed, but I couldn't find if this was
> > ever discussed.
>
> My understanding is that it's permissible to send a length-0 buffer in
> the IPC format for the validity bitmap in the event that the values
> are all null. The receiver may decide to allocate a bitmap of all set
> bits if they need that. I think Java needs this, but I am not sure if
> this case has been implemented and tested (it should be if it is not)
>
This should read "in the event that the value are all _not_ null"
> In principle I don't see an issue with having an analogous
> optimization for all-null arrays with the data buffer, but no
> implementation AFAIK allows for this currently. This probably would
> merit some discussion and probably a change to the format documents if
> we wanted to allow this.
>
> As a high level guiding principle, I don't see value in sending ersatz
> buffers on the wire whose contents are never used. Similarly, creating
> a validity bitmap of all 1's unnecessarily, which adds overhead in the
> case where known non-nullable data is being processed.
>
> The downside is that receivers may need to sanitize a null / length-0
> buffer in such cases or have a special case in some algorithms
>
> - Wes
>
> >
> > A quick perusal of the spec seems to imply they are required (I couldn't
> > find an exception). But I just wanted to confirm this was the case.
> >
> > Thanks,
> > Micah
Re: [Format] For all null arrays are data-buffers required?
Posted by Wes McKinney <we...@gmail.com>.
hi Micah,
I think it's most productive to view things through the lens of the
binary protocol (i.e. the IPC/RPC wire format)
On Wed, Mar 27, 2019 at 1:59 AM Micah Kornfield <em...@gmail.com> wrote:
>
> Similar to how there is no validity-buffer required in the format if
> null_count == 0, is a similar optimization for the "data buffer" allowed
> when (null_count == array length)? It seems that if all values are null,
> no data element should ever be accessed, but I couldn't find if this was
> ever discussed.
My understanding is that it's permissible to send a length-0 buffer in
the IPC format for the validity bitmap in the event that the values
are all null. The receiver may decide to allocate a bitmap of all set
bits if they need that. I think Java needs this, but I am not sure if
this case has been implemented and tested (it should be if it is not)
In principle I don't see an issue with having an analogous
optimization for all-null arrays with the data buffer, but no
implementation AFAIK allows for this currently. This probably would
merit some discussion and probably a change to the format documents if
we wanted to allow this.
As a high level guiding principle, I don't see value in sending ersatz
buffers on the wire whose contents are never used. Similarly, creating
a validity bitmap of all 1's unnecessarily, which adds overhead in the
case where known non-nullable data is being processed.
The downside is that receivers may need to sanitize a null / length-0
buffer in such cases or have a special case in some algorithms
- Wes
>
> A quick perusal of the spec seems to imply they are required (I couldn't
> find an exception). But I just wanted to confirm this was the case.
>
> Thanks,
> Micah