Standard/Reserved Metadata keys

[Michael Lavina <mi...@factset.com>]

Hey Team,

I noticed that under extension type the Arrow docs specifically call out to `ARROW:extension:name`  and `ARROW:extension:metadata` as recommended/reserved metadata keys to handle extension types.

2 quick questions


  1.  Do the standard libraries handle that metadata key automatically ?
  2.  Are there other standard or best practice metadata keys that either people generally use or the standard libraries expect?

Thank you,
Michael

Re: Standard/Reserved Metadata keys

[Joris Van den Bossche <jo...@gmail.com>]

On Thu, 8 Apr 2021 at 19:52, Micah Kornfield <em...@gmail.com> wrote:

>  1.  Do the standard libraries handle that metadata key automatically ?
>
> C++, Python and Java have facilities to support them automatically
> (extensions needs to register themselves), I'm not sure about other
> languages.
>
> 2. Are there other standard or best practice metadata keys that either
> people generally use or the standard libraries expect?
>
> I think there are some things related to Pandas types


Indeed, a "pandas" key is used for this. A description of the information
stored can be found at
https://pandas.pydata.org/docs/dev/development/developer.html#storing-pandas-dataframe-objects-in-apache-parquet-format


> and parquet data that
> are propagated automatically through metadata.


For Parquet, we store the schema in an "ARROW:schema" key in the Parquet
metadata.
When reading back, that key will typically be dropped and not included in
the resulting  Arrow table's schema metadata, though.


> Rust was looking into
> including file sources but I'm not sure that is being externalized.  There
> might be other things, hopefully people will chime in.
>
> On Thu, Apr 8, 2021 at 10:47 AM Michael Lavina <michael.lavina@factset.com
> >
> wrote:
>
> > Hey Team,
> >
> > I noticed that under extension type the Arrow docs specifically call out
> > to `ARROW:extension:name`  and `ARROW:extension:metadata` as
> > recommended/reserved metadata keys to handle extension types.
> >
> > 2 quick questions
> >
> >
> >   1.  Do the standard libraries handle that metadata key automatically ?
> >   2.  Are there other standard or best practice metadata keys that either
> > people generally use or the standard libraries expect?
> >
> > Thank you,
> > Michael
> >
>

Re: Standard/Reserved Metadata keys

[Neal Richardson <ne...@gmail.com>]

Like Pandas, in the R package we store R-specific metadata in the schema;
this lets us preserve all R class and other attributes when round-tripping
data to Feather or Parquet. These are not stored in a way that is readable
or useful to other implementations, partly by design. If there were type
information/metadata that was meaningful across libraries, it would make
sense to define (and document) an extension type.

Some discussion here:
https://arrow.apache.org/docs/r/articles/arrow.html#r-object-attributes

Neal

On Thu, Apr 8, 2021 at 10:52 AM Micah Kornfield <em...@gmail.com>
wrote:

>  1.  Do the standard libraries handle that metadata key automatically ?
>
> C++, Python and Java have facilities to support them automatically
> (extensions needs to register themselves), I'm not sure about other
> languages.
>
> 2. Are there other standard or best practice metadata keys that either
> people generally use or the standard libraries expect?
>
> I think there are some things related to Pandas types and parquet data that
> are propagated automatically through metadata.  Rust was looking into
> including file sources but I'm not sure that is being externalized.  There
> might be other things, hopefully people will chime in.
>
> On Thu, Apr 8, 2021 at 10:47 AM Michael Lavina <michael.lavina@factset.com
> >
> wrote:
>
> > Hey Team,
> >
> > I noticed that under extension type the Arrow docs specifically call out
> > to `ARROW:extension:name`  and `ARROW:extension:metadata` as
> > recommended/reserved metadata keys to handle extension types.
> >
> > 2 quick questions
> >
> >
> >   1.  Do the standard libraries handle that metadata key automatically ?
> >   2.  Are there other standard or best practice metadata keys that either
> > people generally use or the standard libraries expect?
> >
> > Thank you,
> > Michael
> >
>

Re: Standard/Reserved Metadata keys

[Micah Kornfield <em...@gmail.com>]

 1.  Do the standard libraries handle that metadata key automatically ?

C++, Python and Java have facilities to support them automatically
(extensions needs to register themselves), I'm not sure about other
languages.

2. Are there other standard or best practice metadata keys that either
people generally use or the standard libraries expect?

I think there are some things related to Pandas types and parquet data that
are propagated automatically through metadata.  Rust was looking into
including file sources but I'm not sure that is being externalized.  There
might be other things, hopefully people will chime in.

On Thu, Apr 8, 2021 at 10:47 AM Michael Lavina <mi...@factset.com>
wrote:

> Hey Team,
>
> I noticed that under extension type the Arrow docs specifically call out
> to `ARROW:extension:name`  and `ARROW:extension:metadata` as
> recommended/reserved metadata keys to handle extension types.
>
> 2 quick questions
>
>
>   1.  Do the standard libraries handle that metadata key automatically ?
>   2.  Are there other standard or best practice metadata keys that either
> people generally use or the standard libraries expect?
>
> Thank you,
> Michael
>