You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Sebastien Binet <bi...@cern.ch> on 2018/06/18 09:33:27 UTC
Struct support for arrow/go
hi there,
(apologies if this isn't the correct forum to raise this kind of
question...)
I am investigating whether using arrow/go for my application would be
feasible.
I'd like to be able to provide Arrow support (in Go) for reading ROOT
files[1], a file format used in particle physics.
ROOT provides access to row- and col-wise data, with metadata that allows
files to be self-describing.
after some initial investigation of the feature set of the Go package, it
seems it's missing support for Structs.
does anyone know what amount of work adding it to arrow/go would entail?
(I am a reasonably seasoned Go programmer, but a complete newbie with
Arrow...)
or perhaps somebody is already working on that?
cheers,
-s
[1] https://root.cern
Re: Struct support for arrow/go
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
Hello Jim and Sebastien,
> As far as I know, List<Struct<...>> hasn't been implemented yet. These are
> some previous discussions on the Arrow and Parquet mailing lists.
This may be a statement that lead to a bit of confusion. I want to clarify a bit here:
* Arrow C++/Python supports dealing with List<Struct<..>>.
* The Parquet C++ Arrow interface can either deal with List<..> or Struct<..> be they cannot be mixed at the moment in a single type. This limitation is only due to the implementation of the Parquet<->Arrow interface. Arrow itself can deal with these types and Parquet also supports these types. Just the connection between both currently misses the necessary translation code.
> but the Go package, AFAICT, is even missing support for structs (and it's a
> pure-Go package, reimplementing things from first principles).
> hence my question on the amount of work needed to implement (at least)
> support for structs.
As the C++ maintainer, I would say that implementing struct is not extremely complicated. My Go experience is rather limited and I'm not aware what the situation of templates or generics is in Go. Struct types will depend on the fact that you would implement certain things for the Struct itself but mainly have some generic delegation code that calls the underlying primitive types.
Uwe
On Tue, Jun 19, 2018, at 9:21 AM, Sebastien Binet wrote:
> hi Jim,
>
> (happy to see you there!)
>
> On Mon, Jun 18, 2018 at 4:43 PM Jim Pivarski <jp...@gmail.com> wrote:
>
> >
> > -
> >
> > http://mail-archives.apache.org/mod_mbox/parquet-dev/201801.mbox/%3C4221159.JjGKrUjAfh@gudok6%3E
> > -
> >
> > https://lists.apache.org/thread.html/45179fd4023764f8303c6dee57a9b65afcb660ab0c5d60842caf30d9@%3Cdev.arrow.apache.org%3E
> > -
> >
> > http://mail-archives.apache.org/mod_mbox/arrow-dev/201805.mbox/%3CCAJEf=X6OUiAgP5LnFXffteM_Yry0jXazu7ysFv9QH9V14y_bSQ@mail.gmail.com%3E
> >
> >
> thanks for the pointers.
>
> -s
Re: Struct support for arrow/go
Posted by Sebastien Binet <bi...@cern.ch>.
hi Jim,
(happy to see you there!)
On Mon, Jun 18, 2018 at 4:43 PM Jim Pivarski <jp...@gmail.com> wrote:
> As far as I know, List<Struct<...>> hasn't been implemented yet. These are
> some previous discussions on the Arrow and Parquet mailing lists.
>
> -
>
> http://mail-archives.apache.org/mod_mbox/parquet-dev/201801.mbox/%3C4221159.JjGKrUjAfh@gudok6%3E
> -
>
> https://lists.apache.org/thread.html/45179fd4023764f8303c6dee57a9b65afcb660ab0c5d60842caf30d9@%3Cdev.arrow.apache.org%3E
> -
>
> http://mail-archives.apache.org/mod_mbox/arrow-dev/201805.mbox/%3CCAJEf=X6OUiAgP5LnFXffteM_Yry0jXazu7ysFv9QH9V14y_bSQ@mail.gmail.com%3E
>
>
thanks for the pointers.
but the Go package, AFAICT, is even missing support for structs (and it's a
pure-Go package, reimplementing things from first principles).
hence my question on the amount of work needed to implement (at least)
support for structs.
-s
Re: Struct support for arrow/go
Posted by Jim Pivarski <jp...@gmail.com>.
As far as I know, List<Struct<...>> hasn't been implemented yet. These are
some previous discussions on the Arrow and Parquet mailing lists.
-
http://mail-archives.apache.org/mod_mbox/parquet-dev/201801.mbox/%3C4221159.JjGKrUjAfh@gudok6%3E
-
https://lists.apache.org/thread.html/45179fd4023764f8303c6dee57a9b65afcb660ab0c5d60842caf30d9@%3Cdev.arrow.apache.org%3E
-
http://mail-archives.apache.org/mod_mbox/arrow-dev/201805.mbox/%3CCAJEf=X6OUiAgP5LnFXffteM_Yry0jXazu7ysFv9QH9V14y_bSQ@mail.gmail.com%3E
-- Jim
On Mon, Jun 18, 2018 at 11:34 AM Sebastien Binet <bi...@cern.ch> wrote:
> hi there,
>
> (apologies if this isn't the correct forum to raise this kind of
> question...)
>
> I am investigating whether using arrow/go for my application would be
> feasible.
> I'd like to be able to provide Arrow support (in Go) for reading ROOT
> files[1], a file format used in particle physics.
> ROOT provides access to row- and col-wise data, with metadata that allows
> files to be self-describing.
>
> after some initial investigation of the feature set of the Go package, it
> seems it's missing support for Structs.
> does anyone know what amount of work adding it to arrow/go would entail?
> (I am a reasonably seasoned Go programmer, but a complete newbie with
> Arrow...)
>
> or perhaps somebody is already working on that?
>
> cheers,
> -s
>
> [1] https://root.cern
>