You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2023/10/02 12:53:49 UTC
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Hello,
+1 and thanks for working on this!
There'll probably be some minor comments to the format PR, but those
don't deter from accepting these new layouts into the standard.
Regards
Antoine.
Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> Hello,
>
> I'd like to propose adding ListView and LargeListView arrays to the Arrow
> format.
> Previous discussion in [1][2], columnar format description and flatbuffers
> changes in [3].
>
> There are implementations available in both C++ [4] and Go [5]. I'm working
> on the integration tests which I will push to one of the PR branches before
> they are merged. I've made a graph illustrating how this addition affects,
> in a backwards compatible way, the type predicates and inheritance chain on
> the C++ implementation. [6]
>
> The vote will be open for at least 72 hours not counting the weekend.
>
> [ ] +1 add the proposed ListView and LargeListView types to the Apache
> Arrow format
> [ ] -1 do not add the proposed ListView and LargeListView types to the
> Apache Arrow format
> because...
>
> Sincerely,
> Felipe
>
> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> [3] https://github.com/apache/arrow/pull/37877
> [4] https://github.com/apache/arrow/pull/35345
> [5] https://github.com/apache/arrow/pull/37468
> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
>
[RESULT][VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Matt Topol <zo...@gmail.com>.
Hey All,
We have 5 binding +1 votes, 2 non-binding +1 votes, and no -1 votes, so the
vote passes.
Thanks everyone for participating in this and thanks again to Felipe for
pushing through this and making it happen!
--Matt
On Mon, Oct 2, 2023 at 11:50 PM Felipe Oliveira Carvalho <
felipekde@gmail.com> wrote:
> pola.rs, for instance, uses only the LargeListType to represent their
> logical List type. [1]
>
> —
> Felipe
>
> [1]
>
> https://github.com/pola-rs/polars/blob/e3a491e8e5d8d50f3f5d0420a122a74e1e8194a4/crates/polars-core/src/datatypes/dtype.rs#L238C17-L238C17
>
>
> On Tue, 3 Oct 2023 at 00:22 Micah Kornfield <em...@gmail.com> wrote:
>
> > Sorry to chime in late. In practice I'm not sure how much LargeList is
> > used? Are we doing this just for symmetry purposes? Is there a known
> > use-case for it?
> >
> > On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com>
> wrote:
> >
> > > Should have expanded my messages, i forgot that i already +1'd this
> > >
> > > d'oh! Sorry for the spam!
> > >
> > > --Matt
> > >
> > > On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > > > <r....@googlemail.com.invalid> wrote:
> > > >
> > > >> +1
> > > >>
> > > >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> > > >> >
> > > >> > Hello,
> > > >> >
> > > >> > +1 and thanks for working on this!
> > > >> >
> > > >> > There'll probably be some minor comments to the format PR, but
> those
> > > >> > don't deter from accepting these new layouts into the standard.
> > > >> >
> > > >> > Regards
> > > >> >
> > > >> > Antoine.
> > > >> >
> > > >> >
> > > >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> > > >> >> Hello,
> > > >> >>
> > > >> >> I'd like to propose adding ListView and LargeListView arrays to
> the
> > > >> >> Arrow
> > > >> >> format.
> > > >> >> Previous discussion in [1][2], columnar format description and
> > > >> >> flatbuffers
> > > >> >> changes in [3].
> > > >> >>
> > > >> >> There are implementations available in both C++ [4] and Go [5].
> I'm
> > > >> >> working
> > > >> >> on the integration tests which I will push to one of the PR
> > branches
> > > >> >> before
> > > >> >> they are merged. I've made a graph illustrating how this addition
> > > >> >> affects,
> > > >> >> in a backwards compatible way, the type predicates and
> inheritance
> > > >> >> chain on
> > > >> >> the C++ implementation. [6]
> > > >> >>
> > > >> >> The vote will be open for at least 72 hours not counting the
> > weekend.
> > > >> >>
> > > >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> > > Apache
> > > >> >> Arrow format
> > > >> >> [ ] -1 do not add the proposed ListView and LargeListView types
> to
> > > the
> > > >> >> Apache Arrow format
> > > >> >> because...
> > > >> >>
> > > >> >> Sincerely,
> > > >> >> Felipe
> > > >> >>
> > > >> >> [1]
> > https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> > > >> >> [2]
> > https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> > > >> >> [3] https://github.com/apache/arrow/pull/37877
> > > >> >> [4] https://github.com/apache/arrow/pull/35345
> > > >> >> [5] https://github.com/apache/arrow/pull/37468
> > > >> >> [6]
> > > https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> > > >> >>
> > > >>
> > > >
> > >
> >
>
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Felipe Oliveira Carvalho <fe...@gmail.com>.
pola.rs, for instance, uses only the LargeListType to represent their
logical List type. [1]
—
Felipe
[1]
https://github.com/pola-rs/polars/blob/e3a491e8e5d8d50f3f5d0420a122a74e1e8194a4/crates/polars-core/src/datatypes/dtype.rs#L238C17-L238C17
On Tue, 3 Oct 2023 at 00:22 Micah Kornfield <em...@gmail.com> wrote:
> Sorry to chime in late. In practice I'm not sure how much LargeList is
> used? Are we doing this just for symmetry purposes? Is there a known
> use-case for it?
>
> On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com> wrote:
>
> > Should have expanded my messages, i forgot that i already +1'd this
> >
> > d'oh! Sorry for the spam!
> >
> > --Matt
> >
> > On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > > <r....@googlemail.com.invalid> wrote:
> > >
> > >> +1
> > >>
> > >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> > >> >
> > >> > Hello,
> > >> >
> > >> > +1 and thanks for working on this!
> > >> >
> > >> > There'll probably be some minor comments to the format PR, but those
> > >> > don't deter from accepting these new layouts into the standard.
> > >> >
> > >> > Regards
> > >> >
> > >> > Antoine.
> > >> >
> > >> >
> > >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> > >> >> Hello,
> > >> >>
> > >> >> I'd like to propose adding ListView and LargeListView arrays to the
> > >> >> Arrow
> > >> >> format.
> > >> >> Previous discussion in [1][2], columnar format description and
> > >> >> flatbuffers
> > >> >> changes in [3].
> > >> >>
> > >> >> There are implementations available in both C++ [4] and Go [5]. I'm
> > >> >> working
> > >> >> on the integration tests which I will push to one of the PR
> branches
> > >> >> before
> > >> >> they are merged. I've made a graph illustrating how this addition
> > >> >> affects,
> > >> >> in a backwards compatible way, the type predicates and inheritance
> > >> >> chain on
> > >> >> the C++ implementation. [6]
> > >> >>
> > >> >> The vote will be open for at least 72 hours not counting the
> weekend.
> > >> >>
> > >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> > Apache
> > >> >> Arrow format
> > >> >> [ ] -1 do not add the proposed ListView and LargeListView types to
> > the
> > >> >> Apache Arrow format
> > >> >> because...
> > >> >>
> > >> >> Sincerely,
> > >> >> Felipe
> > >> >>
> > >> >> [1]
> https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> > >> >> [2]
> https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> > >> >> [3] https://github.com/apache/arrow/pull/37877
> > >> >> [4] https://github.com/apache/arrow/pull/35345
> > >> >> [5] https://github.com/apache/arrow/pull/37468
> > >> >> [6]
> > https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> > >> >>
> > >>
> > >
> >
>
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Micah Kornfield <em...@gmail.com>.
Sorry to chime in late. In practice I'm not sure how much LargeList is
used? Are we doing this just for symmetry purposes? Is there a known
use-case for it?
On Mon, Oct 2, 2023 at 11:20 AM Matt Topol <zo...@gmail.com> wrote:
> Should have expanded my messages, i forgot that i already +1'd this
>
> d'oh! Sorry for the spam!
>
> --Matt
>
> On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com> wrote:
>
> > +1
> >
> > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> > <r....@googlemail.com.invalid> wrote:
> >
> >> +1
> >>
> >> On 02/10/2023 13:53, Antoine Pitrou wrote:
> >> >
> >> > Hello,
> >> >
> >> > +1 and thanks for working on this!
> >> >
> >> > There'll probably be some minor comments to the format PR, but those
> >> > don't deter from accepting these new layouts into the standard.
> >> >
> >> > Regards
> >> >
> >> > Antoine.
> >> >
> >> >
> >> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> >> >> Hello,
> >> >>
> >> >> I'd like to propose adding ListView and LargeListView arrays to the
> >> >> Arrow
> >> >> format.
> >> >> Previous discussion in [1][2], columnar format description and
> >> >> flatbuffers
> >> >> changes in [3].
> >> >>
> >> >> There are implementations available in both C++ [4] and Go [5]. I'm
> >> >> working
> >> >> on the integration tests which I will push to one of the PR branches
> >> >> before
> >> >> they are merged. I've made a graph illustrating how this addition
> >> >> affects,
> >> >> in a backwards compatible way, the type predicates and inheritance
> >> >> chain on
> >> >> the C++ implementation. [6]
> >> >>
> >> >> The vote will be open for at least 72 hours not counting the weekend.
> >> >>
> >> >> [ ] +1 add the proposed ListView and LargeListView types to the
> Apache
> >> >> Arrow format
> >> >> [ ] -1 do not add the proposed ListView and LargeListView types to
> the
> >> >> Apache Arrow format
> >> >> because...
> >> >>
> >> >> Sincerely,
> >> >> Felipe
> >> >>
> >> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> >> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> >> >> [3] https://github.com/apache/arrow/pull/37877
> >> >> [4] https://github.com/apache/arrow/pull/35345
> >> >> [5] https://github.com/apache/arrow/pull/37468
> >> >> [6]
> https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> >> >>
> >>
> >
>
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Matt Topol <zo...@gmail.com>.
Should have expanded my messages, i forgot that i already +1'd this
d'oh! Sorry for the spam!
--Matt
On Mon, Oct 2, 2023 at 2:19 PM Matt Topol <zo...@gmail.com> wrote:
> +1
>
> On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
> <r....@googlemail.com.invalid> wrote:
>
>> +1
>>
>> On 02/10/2023 13:53, Antoine Pitrou wrote:
>> >
>> > Hello,
>> >
>> > +1 and thanks for working on this!
>> >
>> > There'll probably be some minor comments to the format PR, but those
>> > don't deter from accepting these new layouts into the standard.
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
>> >> Hello,
>> >>
>> >> I'd like to propose adding ListView and LargeListView arrays to the
>> >> Arrow
>> >> format.
>> >> Previous discussion in [1][2], columnar format description and
>> >> flatbuffers
>> >> changes in [3].
>> >>
>> >> There are implementations available in both C++ [4] and Go [5]. I'm
>> >> working
>> >> on the integration tests which I will push to one of the PR branches
>> >> before
>> >> they are merged. I've made a graph illustrating how this addition
>> >> affects,
>> >> in a backwards compatible way, the type predicates and inheritance
>> >> chain on
>> >> the C++ implementation. [6]
>> >>
>> >> The vote will be open for at least 72 hours not counting the weekend.
>> >>
>> >> [ ] +1 add the proposed ListView and LargeListView types to the Apache
>> >> Arrow format
>> >> [ ] -1 do not add the proposed ListView and LargeListView types to the
>> >> Apache Arrow format
>> >> because...
>> >>
>> >> Sincerely,
>> >> Felipe
>> >>
>> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
>> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
>> >> [3] https://github.com/apache/arrow/pull/37877
>> >> [4] https://github.com/apache/arrow/pull/35345
>> >> [5] https://github.com/apache/arrow/pull/37468
>> >> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
>> >>
>>
>
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Matt Topol <zo...@gmail.com>.
+1
On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies
<r....@googlemail.com.invalid> wrote:
> +1
>
> On 02/10/2023 13:53, Antoine Pitrou wrote:
> >
> > Hello,
> >
> > +1 and thanks for working on this!
> >
> > There'll probably be some minor comments to the format PR, but those
> > don't deter from accepting these new layouts into the standard.
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
> >> Hello,
> >>
> >> I'd like to propose adding ListView and LargeListView arrays to the
> >> Arrow
> >> format.
> >> Previous discussion in [1][2], columnar format description and
> >> flatbuffers
> >> changes in [3].
> >>
> >> There are implementations available in both C++ [4] and Go [5]. I'm
> >> working
> >> on the integration tests which I will push to one of the PR branches
> >> before
> >> they are merged. I've made a graph illustrating how this addition
> >> affects,
> >> in a backwards compatible way, the type predicates and inheritance
> >> chain on
> >> the C++ implementation. [6]
> >>
> >> The vote will be open for at least 72 hours not counting the weekend.
> >>
> >> [ ] +1 add the proposed ListView and LargeListView types to the Apache
> >> Arrow format
> >> [ ] -1 do not add the proposed ListView and LargeListView types to the
> >> Apache Arrow format
> >> because...
> >>
> >> Sincerely,
> >> Felipe
> >>
> >> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> >> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> >> [3] https://github.com/apache/arrow/pull/37877
> >> [4] https://github.com/apache/arrow/pull/35345
> >> [5] https://github.com/apache/arrow/pull/37468
> >> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> >>
>
Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format
Posted by Raphael Taylor-Davies <r....@googlemail.com.INVALID>.
+1
On 02/10/2023 13:53, Antoine Pitrou wrote:
>
> Hello,
>
> +1 and thanks for working on this!
>
> There'll probably be some minor comments to the format PR, but those
> don't deter from accepting these new layouts into the standard.
>
> Regards
>
> Antoine.
>
>
> Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit :
>> Hello,
>>
>> I'd like to propose adding ListView and LargeListView arrays to the
>> Arrow
>> format.
>> Previous discussion in [1][2], columnar format description and
>> flatbuffers
>> changes in [3].
>>
>> There are implementations available in both C++ [4] and Go [5]. I'm
>> working
>> on the integration tests which I will push to one of the PR branches
>> before
>> they are merged. I've made a graph illustrating how this addition
>> affects,
>> in a backwards compatible way, the type predicates and inheritance
>> chain on
>> the C++ implementation. [6]
>>
>> The vote will be open for at least 72 hours not counting the weekend.
>>
>> [ ] +1 add the proposed ListView and LargeListView types to the Apache
>> Arrow format
>> [ ] -1 do not add the proposed ListView and LargeListView types to the
>> Apache Arrow format
>> because...
>>
>> Sincerely,
>> Felipe
>>
>> [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
>> [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
>> [3] https://github.com/apache/arrow/pull/37877
>> [4] https://github.com/apache/arrow/pull/35345
>> [5] https://github.com/apache/arrow/pull/37468
>> [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
>>