You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2023/10/02 12:53:49 UTC

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Hello,

+1 and thanks for working on this!

There'll probably be some minor comments to the format PR, but those 
don't deter from accepting these new layouts into the standard.

Regards

Antoine.


Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> Hello,
> 
> I'd like to propose adding ListView and LargeListView arrays to the Arrow
> format.
> Previous discussion in [1][2], columnar format description and flatbuffers
> changes in [3].
> 
> There are implementations available in both C++ [4] and Go [5]. I'm working
> on the integration tests which I will push to one of the PR branches before
> they are merged. I've made a graph illustrating how this addition affects,
> in a backwards compatible way, the type predicates and inheritance chain on
> the C++ implementation. [6]
> 
> The vote will be open for at least 72 hours not counting the weekend.
> 
> [ ] +1 add the proposed ListView and LargeListView types to the Apache
> Arrow format
> [ ] -1 do not add the proposed ListView and LargeListView types to the
> Apache Arrow format
> because...
> 
> Sincerely,
> Felipe
> 
> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> [3] https://github.com/apache/arrow/pull/37877
> [4] https://github.com/apache/arrow/pull/35345
> [5] https://github.com/apache/arrow/pull/37468
> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> 

[RESULT][VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Matt Topol <zo...@gmail.com>.
Hey All,

We have 5 binding +1 votes, 2 non-binding +1 votes, and no -1 votes, so the
vote passes.

Thanks everyone for participating in this and thanks again to Felipe for
pushing through this and making it happen!

--Matt

On Mon, Oct 2, 2023 at 11:50 PM Felipe Oliveira Carvalho <
felipekde@gmail.com> wrote:

> pola.rs, for instance, uses only the LargeListType to represent their
> logical List type. [1]
>
> —
> Felipe
>
> [1]
>
> https://github.com/pola-rs/polars/blob/e3a491e8e5d8d50f3f5d0420a122a74e1e8194a4/crates/polars-core/src/datatypes/dtype.rs#L238C17-L238C17
>
>
> On Tue, 3 Oct 2023 at 00:22 Micah Kornfield <em...@gmail.com> wrote:
>
> > Sorry to chime in late.  In practice I'm not sure how much LargeList is
> > used?  Are we doing this just for symmetry purposes?  Is there a known
> > use-case for it?
> >
> > On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com>
> wrote:
> >
> > > Should have expanded my messages, i forgot that i already +1'd this
> > >
> > > d'oh! Sorry for the spam!
> > >
> > > --Matt
> > >
> > > On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > > > <r....@googlemail.com.invalid> wrote:
> > > >
> > > >> +1
> > > >>
> > > >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> > > >> >
> > > >> > Hello,
> > > >> >
> > > >> > +1 and thanks for working on this!
> > > >> >
> > > >> > There'll probably be some minor comments to the format PR, but
> those
> > > >> > don't deter from accepting these new layouts into the standard.
> > > >> >
> > > >> > Regards
> > > >> >
> > > >> > Antoine.
> > > >> >
> > > >> >
> > > >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> > > >> >> Hello,
> > > >> >>
> > > >> >> I'd like to propose adding ListView and LargeListView arrays to
> the
> > > >> >> Arrow
> > > >> >> format.
> > > >> >> Previous discussion in [1][2], columnar format description and
> > > >> >> flatbuffers
> > > >> >> changes in [3].
> > > >> >>
> > > >> >> There are implementations available in both C++ [4] and Go [5].
> I'm
> > > >> >> working
> > > >> >> on the integration tests which I will push to one of the PR
> > branches
> > > >> >> before
> > > >> >> they are merged. I've made a graph illustrating how this addition
> > > >> >> affects,
> > > >> >> in a backwards compatible way, the type predicates and
> inheritance
> > > >> >> chain on
> > > >> >> the C++ implementation. [6]
> > > >> >>
> > > >> >> The vote will be open for at least 72 hours not counting the
> > weekend.
> > > >> >>
> > > >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> > > Apache
> > > >> >> Arrow format
> > > >> >> [ ] -1 do not add the proposed ListView and LargeListView types
> to
> > > the
> > > >> >> Apache Arrow format
> > > >> >> because...
> > > >> >>
> > > >> >> Sincerely,
> > > >> >> Felipe
> > > >> >>
> > > >> >> [1]
> > https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> > > >> >> [2]
> > https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> > > >> >> [3] https://github.com/apache/arrow/pull/37877
> > > >> >> [4] https://github.com/apache/arrow/pull/35345
> > > >> >> [5] https://github.com/apache/arrow/pull/37468
> > > >> >> [6]
> > > https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> > > >> >>
> > > >>
> > > >
> > >
> >
>

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Felipe Oliveira Carvalho <fe...@gmail.com>.
pola.rs, for instance, uses only the LargeListType to represent their
logical List type. [1]

—
Felipe

[1]
https://github.com/pola-rs/polars/blob/e3a491e8e5d8d50f3f5d0420a122a74e1e8194a4/crates/polars-core/src/datatypes/dtype.rs#L238C17-L238C17


On Tue, 3 Oct 2023 at 00:22 Micah Kornfield <em...@gmail.com> wrote:

> Sorry to chime in late.  In practice I'm not sure how much LargeList is
> used?  Are we doing this just for symmetry purposes?  Is there a known
> use-case for it?
>
> On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com> wrote:
>
> > Should have expanded my messages, i forgot that i already +1'd this
> >
> > d'oh! Sorry for the spam!
> >
> > --Matt
> >
> > On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > > <r....@googlemail.com.invalid> wrote:
> > >
> > >> +1
> > >>
> > >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> > >> >
> > >> > Hello,
> > >> >
> > >> > +1 and thanks for working on this!
> > >> >
> > >> > There'll probably be some minor comments to the format PR, but those
> > >> > don't deter from accepting these new layouts into the standard.
> > >> >
> > >> > Regards
> > >> >
> > >> > Antoine.
> > >> >
> > >> >
> > >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> > >> >> Hello,
> > >> >>
> > >> >> I'd like to propose adding ListView and LargeListView arrays to the
> > >> >> Arrow
> > >> >> format.
> > >> >> Previous discussion in [1][2], columnar format description and
> > >> >> flatbuffers
> > >> >> changes in [3].
> > >> >>
> > >> >> There are implementations available in both C++ [4] and Go [5]. I'm
> > >> >> working
> > >> >> on the integration tests which I will push to one of the PR
> branches
> > >> >> before
> > >> >> they are merged. I've made a graph illustrating how this addition
> > >> >> affects,
> > >> >> in a backwards compatible way, the type predicates and inheritance
> > >> >> chain on
> > >> >> the C++ implementation. [6]
> > >> >>
> > >> >> The vote will be open for at least 72 hours not counting the
> weekend.
> > >> >>
> > >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> > Apache
> > >> >> Arrow format
> > >> >> [ ] -1 do not add the proposed ListView and LargeListView types to
> > the
> > >> >> Apache Arrow format
> > >> >> because...
> > >> >>
> > >> >> Sincerely,
> > >> >> Felipe
> > >> >>
> > >> >> [1]
> https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> > >> >> [2]
> https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> > >> >> [3] https://github.com/apache/arrow/pull/37877
> > >> >> [4] https://github.com/apache/arrow/pull/35345
> > >> >> [5] https://github.com/apache/arrow/pull/37468
> > >> >> [6]
> > https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> > >> >>
> > >>
> > >
> >
>

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Micah Kornfield <em...@gmail.com>.
Sorry to chime in late.  In practice I'm not sure how much LargeList is
used?  Are we doing this just for symmetry purposes?  Is there a known
use-case for it?

On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com> wrote:

> Should have expanded my messages, i forgot that i already +1'd this
>
> d'oh! Sorry for the spam!
>
> --Matt
>
> On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com> wrote:
>
> > +1
> >
> > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > <r....@googlemail.com.invalid> wrote:
> >
> >> +1
> >>
> >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> >> >
> >> > Hello,
> >> >
> >> > +1 and thanks for working on this!
> >> >
> >> > There'll probably be some minor comments to the format PR, but those
> >> > don't deter from accepting these new layouts into the standard.
> >> >
> >> > Regards
> >> >
> >> > Antoine.
> >> >
> >> >
> >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> >> >> Hello,
> >> >>
> >> >> I'd like to propose adding ListView and LargeListView arrays to the
> >> >> Arrow
> >> >> format.
> >> >> Previous discussion in [1][2], columnar format description and
> >> >> flatbuffers
> >> >> changes in [3].
> >> >>
> >> >> There are implementations available in both C++ [4] and Go [5]. I'm
> >> >> working
> >> >> on the integration tests which I will push to one of the PR branches
> >> >> before
> >> >> they are merged. I've made a graph illustrating how this addition
> >> >> affects,
> >> >> in a backwards compatible way, the type predicates and inheritance
> >> >> chain on
> >> >> the C++ implementation. [6]
> >> >>
> >> >> The vote will be open for at least 72 hours not counting the weekend.
> >> >>
> >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> Apache
> >> >> Arrow format
> >> >> [ ] -1 do not add the proposed ListView and LargeListView types to
> the
> >> >> Apache Arrow format
> >> >> because...
> >> >>
> >> >> Sincerely,
> >> >> Felipe
> >> >>
> >> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> >> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> >> >> [3] https://github.com/apache/arrow/pull/37877
> >> >> [4] https://github.com/apache/arrow/pull/35345
> >> >> [5] https://github.com/apache/arrow/pull/37468
> >> >> [6]
> https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> >> >>
> >>
> >
>

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Matt Topol <zo...@gmail.com>.
Should have expanded my messages, i forgot that i already +1'd this

d'oh! Sorry for the spam!

--Matt

On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com> wrote:

> +1
>
> On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> <r....@googlemail.com.invalid> wrote:
>
>> +1
>>
>> On 02/10/2023 13:53, Antoine Pitrou wrote:
>> >
>> > Hello,
>> >
>> > +1 and thanks for working on this!
>> >
>> > There'll probably be some minor comments to the format PR, but those
>> > don't deter from accepting these new layouts into the standard.
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
>> >> Hello,
>> >>
>> >> I'd like to propose adding ListView and LargeListView arrays to the
>> >> Arrow
>> >> format.
>> >> Previous discussion in [1][2], columnar format description and
>> >> flatbuffers
>> >> changes in [3].
>> >>
>> >> There are implementations available in both C++ [4] and Go [5]. I'm
>> >> working
>> >> on the integration tests which I will push to one of the PR branches
>> >> before
>> >> they are merged. I've made a graph illustrating how this addition
>> >> affects,
>> >> in a backwards compatible way, the type predicates and inheritance
>> >> chain on
>> >> the C++ implementation. [6]
>> >>
>> >> The vote will be open for at least 72 hours not counting the weekend.
>> >>
>> >> [ ] +1 add the proposed ListView and LargeListView types to the Apache
>> >> Arrow format
>> >> [ ] -1 do not add the proposed ListView and LargeListView types to the
>> >> Apache Arrow format
>> >> because...
>> >>
>> >> Sincerely,
>> >> Felipe
>> >>
>> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
>> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
>> >> [3] https://github.com/apache/arrow/pull/37877
>> >> [4] https://github.com/apache/arrow/pull/35345
>> >> [5] https://github.com/apache/arrow/pull/37468
>> >> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
>> >>
>>
>

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Matt Topol <zo...@gmail.com>.
+1

On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
<r....@googlemail.com.invalid> wrote:

> +1
>
> On 02/10/2023 13:53, Antoine Pitrou wrote:
> >
> > Hello,
> >
> > +1 and thanks for working on this!
> >
> > There'll probably be some minor comments to the format PR, but those
> > don't deter from accepting these new layouts into the standard.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> >> Hello,
> >>
> >> I'd like to propose adding ListView and LargeListView arrays to the
> >> Arrow
> >> format.
> >> Previous discussion in [1][2], columnar format description and
> >> flatbuffers
> >> changes in [3].
> >>
> >> There are implementations available in both C++ [4] and Go [5]. I'm
> >> working
> >> on the integration tests which I will push to one of the PR branches
> >> before
> >> they are merged. I've made a graph illustrating how this addition
> >> affects,
> >> in a backwards compatible way, the type predicates and inheritance
> >> chain on
> >> the C++ implementation. [6]
> >>
> >> The vote will be open for at least 72 hours not counting the weekend.
> >>
> >> [ ] +1 add the proposed ListView and LargeListView types to the Apache
> >> Arrow format
> >> [ ] -1 do not add the proposed ListView and LargeListView types to the
> >> Apache Arrow format
> >> because...
> >>
> >> Sincerely,
> >> Felipe
> >>
> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> >> [3] https://github.com/apache/arrow/pull/37877
> >> [4] https://github.com/apache/arrow/pull/35345
> >> [5] https://github.com/apache/arrow/pull/37468
> >> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> >>
>

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

Posted by Raphael Taylor-Davies <r....@googlemail.com.INVALID>.
+1

On 02/10/2023 13:53, Antoine Pitrou wrote:
>
> Hello,
>
> +1 and thanks for working on this!
>
> There'll probably be some minor comments to the format PR, but those 
> don't deter from accepting these new layouts into the standard.
>
> Regards
>
> Antoine.
>
>
> Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
>> Hello,
>>
>> I'd like to propose adding ListView and LargeListView arrays to the 
>> Arrow
>> format.
>> Previous discussion in [1][2], columnar format description and 
>> flatbuffers
>> changes in [3].
>>
>> There are implementations available in both C++ [4] and Go [5]. I'm 
>> working
>> on the integration tests which I will push to one of the PR branches 
>> before
>> they are merged. I've made a graph illustrating how this addition 
>> affects,
>> in a backwards compatible way, the type predicates and inheritance 
>> chain on
>> the C++ implementation. [6]
>>
>> The vote will be open for at least 72 hours not counting the weekend.
>>
>> [ ] +1 add the proposed ListView and LargeListView types to the Apache
>> Arrow format
>> [ ] -1 do not add the proposed ListView and LargeListView types to the
>> Apache Arrow format
>> because...
>>
>> Sincerely,
>> Felipe
>>
>> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
>> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
>> [3] https://github.com/apache/arrow/pull/37877
>> [4] https://github.com/apache/arrow/pull/35345
>> [5] https://github.com/apache/arrow/pull/37468
>> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
>>