You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Jorge Cardoso Leitão <jo...@gmail.com> on 2021/08/13 16:48:46 UTC
[Question] what is the purpose of the typeids in the UnionArray?
Hi,
In the UnionArray, there is a level of indirection between types (buffer of
i8s) -> typeId (i8) -> field. For example, the generated_union part of our
integration tests has the data:
types: [5, 5, 5, 5, 7, 7, 7, 7, 5, 5, 7] (len = 11)
typeids: [5, 7]
fields: [int32, utf8]
My understanding is that, to get the field of item 4, we read types[4] (7),
look for the index of it in typeids (1), and take the field of index 1
(utf8), and then read the value (4 or other depending on sparsess).
Does someone know the rationale for the intermediare typeid? I.e. couldn't
the types contain the index of the field directly [0, 0, 0, 0, 1, 1, 1, 1,
0, 0,1] (replace 5 by 0, 7 by 1, and not use typeids)?
Best,
Jorge
Re: [Question] what is the purpose of the typeids in the UnionArray?
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
Thanks a lot for the confirmation, Micah.
It still seems possible to remove the typeid without loss of generality
(not that I am advocating for), as all fields are declared as children of
the field, and it is thus possible to declare fields that the union does
not currently contain in the metadata, which are mapped to empty ArrayData
children (in dense).
A minor annoyance is that the typeid does not allow a zero-copy over the C
data interface, as we need to initialize an intermediary hashmap to make
the lookup fast, or take the penalty hit to search (linearly over the
number of fields) over the typeids for the type id.
Best,
Jorge
On Fri, Aug 13, 2021 at 7:07 PM Micah Kornfield <em...@gmail.com>
wrote:
> Jorge,
> I think your analysis is correct. Some historical context on why there is
> an indication is covered on the original JIRA:
> https://issues.apache.org/jira/browse/ARROW-257
>
> Some other discussions:
>
> https://lists.apache.org/x/thread.html/75028183d54cb4f6ff588b043fe126f10b2cba8e373673fad6ba889d@%3Cdev.arrow.apache.org%3E
>
> https://lists.apache.org/x/thread.html/b219ef51dda71bef83dcdec94e68e2881d49f751b29a8c1251f653d5@%3Cdev.arrow.apache.org%3E
>
> -Micah
>
> On Fri, Aug 13, 2021 at 10:57 AM Keith Kraus <ke...@gmail.com>
> wrote:
>
> > How would using the typeid directly work with arbitrary Extension types?
> >
> > -Keith
> >
> > On Fri, Aug 13, 2021 at 12:49 PM Jorge Cardoso Leitão <
> > jorgecarleitao@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > In the UnionArray, there is a level of indirection between types
> (buffer
> > of
> > > i8s) -> typeId (i8) -> field. For example, the generated_union part of
> > our
> > > integration tests has the data:
> > >
> > > types: [5, 5, 5, 5, 7, 7, 7, 7, 5, 5, 7] (len = 11)
> > > typeids: [5, 7]
> > > fields: [int32, utf8]
> > >
> > > My understanding is that, to get the field of item 4, we read types[4]
> > (7),
> > > look for the index of it in typeids (1), and take the field of index 1
> > > (utf8), and then read the value (4 or other depending on sparsess).
> > >
> > > Does someone know the rationale for the intermediare typeid? I.e.
> > couldn't
> > > the types contain the index of the field directly [0, 0, 0, 0, 1, 1, 1,
> > 1,
> > > 0, 0,1] (replace 5 by 0, 7 by 1, and not use typeids)?
> > >
> > > Best,
> > > Jorge
> > >
> >
>
Re: [Question] what is the purpose of the typeids in the UnionArray?
Posted by Micah Kornfield <em...@gmail.com>.
Jorge,
I think your analysis is correct. Some historical context on why there is
an indication is covered on the original JIRA:
https://issues.apache.org/jira/browse/ARROW-257
Some other discussions:
https://lists.apache.org/x/thread.html/75028183d54cb4f6ff588b043fe126f10b2cba8e373673fad6ba889d@%3Cdev.arrow.apache.org%3E
https://lists.apache.org/x/thread.html/b219ef51dda71bef83dcdec94e68e2881d49f751b29a8c1251f653d5@%3Cdev.arrow.apache.org%3E
-Micah
On Fri, Aug 13, 2021 at 10:57 AM Keith Kraus <ke...@gmail.com>
wrote:
> How would using the typeid directly work with arbitrary Extension types?
>
> -Keith
>
> On Fri, Aug 13, 2021 at 12:49 PM Jorge Cardoso Leitão <
> jorgecarleitao@gmail.com> wrote:
>
> > Hi,
> >
> > In the UnionArray, there is a level of indirection between types (buffer
> of
> > i8s) -> typeId (i8) -> field. For example, the generated_union part of
> our
> > integration tests has the data:
> >
> > types: [5, 5, 5, 5, 7, 7, 7, 7, 5, 5, 7] (len = 11)
> > typeids: [5, 7]
> > fields: [int32, utf8]
> >
> > My understanding is that, to get the field of item 4, we read types[4]
> (7),
> > look for the index of it in typeids (1), and take the field of index 1
> > (utf8), and then read the value (4 or other depending on sparsess).
> >
> > Does someone know the rationale for the intermediare typeid? I.e.
> couldn't
> > the types contain the index of the field directly [0, 0, 0, 0, 1, 1, 1,
> 1,
> > 0, 0,1] (replace 5 by 0, 7 by 1, and not use typeids)?
> >
> > Best,
> > Jorge
> >
>
Re: [Question] what is the purpose of the typeids in the UnionArray?
Posted by Keith Kraus <ke...@gmail.com>.
How would using the typeid directly work with arbitrary Extension types?
-Keith
On Fri, Aug 13, 2021 at 12:49 PM Jorge Cardoso Leitão <
jorgecarleitao@gmail.com> wrote:
> Hi,
>
> In the UnionArray, there is a level of indirection between types (buffer of
> i8s) -> typeId (i8) -> field. For example, the generated_union part of our
> integration tests has the data:
>
> types: [5, 5, 5, 5, 7, 7, 7, 7, 5, 5, 7] (len = 11)
> typeids: [5, 7]
> fields: [int32, utf8]
>
> My understanding is that, to get the field of item 4, we read types[4] (7),
> look for the index of it in typeids (1), and take the field of index 1
> (utf8), and then read the value (4 or other depending on sparsess).
>
> Does someone know the rationale for the intermediare typeid? I.e. couldn't
> the types contain the index of the field directly [0, 0, 0, 0, 1, 1, 1, 1,
> 0, 0,1] (replace 5 by 0, 7 by 1, and not use typeids)?
>
> Best,
> Jorge
>