You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Jim Pivarski <jp...@gmail.com> on 2016/08/03 15:00:02 UTC
Parquet for High Energy Physics
Hi,
I'd like to use parquet-cpp for High Energy Physics (HEP) and possibly
contribute to the core to support that use-case, but I'm having trouble
determining the status of the C++ project.
Most HEP data is stored in the ROOT file format (
https://root.cern.ch/root/InputOutput.html), which represents complex,
nested, cross-referenced C++ objects with a columnar layout so that a
subset of fields can be individually read, individually compressed, and
quickly scanned. I believe that these benefits can be satisfied by Parquet,
with the additional benefit that it's a standard with a specification that
can be read or written in multiple languages. (Parquet can't be used as a
random-writable object database, but this feature of ROOT isn't widely
used.)
To convert between ROOT and Parquet, I would need to implement ROOT's
"StreamerInfo" object schema (https://root.cern.ch/root/SchemaEvolution.html)
into a Logical Type Definition, on par with AvroRecordReader, but also
supporting pointer references (as an Int64 -> object map).
Parquet C++'s TODO (https://github.com/apache/parquet-cpp/blob/master/TODO)
states that this record abstraction, as well as nested schemas and
file-writing, haven't been implemented. However, the TODO is also 2 years
old, where I see a burst of activity this year in GitHub. Is the TODO out
of date?
Will any of the core developers be at KDD16 (http://www.kdd.org/kdd2016/)
or elsewhere in San Francisco on August 15 or 16? If so, could we meet in
person so that we can talk in detail about where the hooks I'm looking for
are and how I can contribute? (Or *when* I should contribute, if there's a
major refactoring in the works.)
Thanks!
-- Jim
Re: Parquet for High Energy Physics
Posted by Julien Le Dem <ju...@ledem.net>.
I’m CC’ing the parquet-cpp main contributors. You are right that there has been a lot of progress recently.
(some of the goals are to provide Python and Vertica access to Parquet files)
I’ll let them comment on the progress.
You’re certainly welcome to contribute.
Julien
> On Aug 3, 2016, at 8:00 AM, Jim Pivarski <jp...@gmail.com> wrote:
>
> Hi,
>
> I'd like to use parquet-cpp for High Energy Physics (HEP) and possibly
> contribute to the core to support that use-case, but I'm having trouble
> determining the status of the C++ project.
>
> Most HEP data is stored in the ROOT file format (
> https://root.cern.ch/root/InputOutput.html), which represents complex,
> nested, cross-referenced C++ objects with a columnar layout so that a
> subset of fields can be individually read, individually compressed, and
> quickly scanned. I believe that these benefits can be satisfied by Parquet,
> with the additional benefit that it's a standard with a specification that
> can be read or written in multiple languages. (Parquet can't be used as a
> random-writable object database, but this feature of ROOT isn't widely
> used.)
>
> To convert between ROOT and Parquet, I would need to implement ROOT's
> "StreamerInfo" object schema (https://root.cern.ch/root/SchemaEvolution.html)
> into a Logical Type Definition, on par with AvroRecordReader, but also
> supporting pointer references (as an Int64 -> object map).
>
> Parquet C++'s TODO (https://github.com/apache/parquet-cpp/blob/master/TODO)
> states that this record abstraction, as well as nested schemas and
> file-writing, haven't been implemented. However, the TODO is also 2 years
> old, where I see a burst of activity this year in GitHub. Is the TODO out
> of date?
>
> Will any of the core developers be at KDD16 (http://www.kdd.org/kdd2016/)
> or elsewhere in San Francisco on August 15 or 16? If so, could we meet in
> person so that we can talk in detail about where the hooks I'm looking for
> are and how I can contribute? (Or *when* I should contribute, if there's a
> major refactoring in the works.)
>
> Thanks!
> -- Jim