You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Anja <an...@gmail.com> on 2023/08/02 16:43:29 UTC
Null buffer spec incompatibility between arrow2 and arrow-C++
Hi!
I opened two issues recently on a spec incompatibility between arrow2 and
arrow-C++: https://github.com/apache/arrow/issues/36960 and
https://github.com/jorgecarleitao/arrow2/issues/1524.
Namely, there are situations where arrow-C++ will legally create a null
buffer, and arrow2 does not allow null buffers.
I am happy to provide code-effort, but I am looking for consensus from the
community on this issue. I think an interoperability issue like this is
important to resolve, especially with the plans to unify arrow2 and
arrow-rs: https://github.com/apache/arrow-rs/issues/1176
The `kNonNullFiller` (
https://github.com/apache/arrow/blob/a06b2618420ef89431373a9e8f07a5da64d546a5/cpp/src/arrow/util/ubsan.h#L33)
exists for being used in cases where null buffers can cause issues. I am
proposing changes such that it gets pointed to for representing the reading
of an empty table.
Are there any thoughts? I welcome engagement on either of the issues. This
problem can, in theory, be solved within either project.
~Anja
Re: Null buffer spec incompatibility between arrow2 and arrow-C++
Posted by Anja <an...@gmail.com>.
Oh, thank you!
It is true that I thought arrow2 was the one that was incorrect, but when I
saw the low activity on recent issues, I assumed it was a project with
lower maintenance capacities. Thank you for pointing out [3]!
On Wed, 2 Aug 2023 at 10:09, Raphael Taylor-Davies
<r....@googlemail.com.invalid> wrote:
> Hi Anja,
>
> I believe this was clarified in [1] with the addition of the following
> text to the standard
>
> > The buffer pointers MAY be null only in two situations:
> >
> > 1. for the null bitmap buffer, if :c:member:`ArrowArray.null_count`
> is 0;
> > 2. for any buffer, if the size in bytes of the corresponding buffer
> would be 0.
>
> This is supported by arrow-rs [2], and I believe arrow2 is incorrect in
> not supporting this. I would recommend taking this up with the arrow2
> maintainers [3].
>
> Kind Regards,
>
> Raphael Taylor-Davies
>
> [1]: https://github.com/apache/arrow/pull/14808
> [2]: https://github.com/apache/arrow-rs/pull/3276
> [3]: https://github.com/jorgecarleitao/arrow2/pull/1476
>
> On 02/08/2023 17:43, Anja wrote:
> > Hi!
> >
> > I opened two issues recently on a spec incompatibility between arrow2 and
> > arrow-C++: https://github.com/apache/arrow/issues/36960 and
> > .
> >
> > Namely, there are situations where arrow-C++ will legally create a null
> > buffer, and arrow2 does not allow null buffers.
> >
> > I am happy to provide code-effort, but I am looking for consensus from
> the
> > community on this issue. I think an interoperability issue like this is
> > important to resolve, especially with the plans to unify arrow2 and
> > arrow-rs: https://github.com/apache/arrow-rs/issues/1176
> >
> > The `kNonNullFiller` (
> >
> https://github.com/apache/arrow/blob/a06b2618420ef89431373a9e8f07a5da64d546a5/cpp/src/arrow/util/ubsan.h#L33
> )
> > exists for being used in cases where null buffers can cause issues. I am
> > proposing changes such that it gets pointed to for representing the
> reading
> > of an empty table.
> >
> > Are there any thoughts? I welcome engagement on either of the issues.
> This
> > problem can, in theory, be solved within either project.
> >
> > ~Anja
> >
>
Re: Null buffer spec incompatibility between arrow2 and arrow-C++
Posted by Raphael Taylor-Davies <r....@googlemail.com.INVALID>.
Hi Anja,
I believe this was clarified in [1] with the addition of the following
text to the standard
> The buffer pointers MAY be null only in two situations:
>
> 1. for the null bitmap buffer, if :c:member:`ArrowArray.null_count`
is 0;
> 2. for any buffer, if the size in bytes of the corresponding buffer
would be 0.
This is supported by arrow-rs [2], and I believe arrow2 is incorrect in
not supporting this. I would recommend taking this up with the arrow2
maintainers [3].
Kind Regards,
Raphael Taylor-Davies
[1]: https://github.com/apache/arrow/pull/14808
[2]: https://github.com/apache/arrow-rs/pull/3276
[3]: https://github.com/jorgecarleitao/arrow2/pull/1476
On 02/08/2023 17:43, Anja wrote:
> Hi!
>
> I opened two issues recently on a spec incompatibility between arrow2 and
> arrow-C++: https://github.com/apache/arrow/issues/36960 and
> .
>
> Namely, there are situations where arrow-C++ will legally create a null
> buffer, and arrow2 does not allow null buffers.
>
> I am happy to provide code-effort, but I am looking for consensus from the
> community on this issue. I think an interoperability issue like this is
> important to resolve, especially with the plans to unify arrow2 and
> arrow-rs: https://github.com/apache/arrow-rs/issues/1176
>
> The `kNonNullFiller` (
> https://github.com/apache/arrow/blob/a06b2618420ef89431373a9e8f07a5da64d546a5/cpp/src/arrow/util/ubsan.h#L33)
> exists for being used in cases where null buffers can cause issues. I am
> proposing changes such that it gets pointed to for representing the reading
> of an empty table.
>
> Are there any thoughts? I welcome engagement on either of the issues. This
> problem can, in theory, be solved within either project.
>
> ~Anja
>