[jira] [Created] (ARROW-12823) [Parquet][Python] Read and write file/column metadata using pandas attrs

["Alan Snow (Jira)" <ji...@apache.org>]

Alan Snow created ARROW-12823:
---------------------------------

             Summary: [Parquet][Python] Read and write file/column metadata using pandas attrs
                 Key: ARROW-12823
                 URL: https://issues.apache.org/jira/browse/ARROW-12823
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Parquet, Python
            Reporter: Alan Snow


Related: https://github.com/pandas-dev/pandas/issues/20521

What the general thoughts are to use [DataFrame.attrs|https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.attrs.html#pandas-dataframe-attrs] and [Series.attrs|https://pandas.pydata.org/pandas-docs/stable//reference/api/pandas.Series.attrs.html#pandas-series-attrs] for reading and writing metadata to/from parquet?

For example, here is how the metadata would be written:
{code:python}
pdf = pandas.DataFrame({"a": [1]})
pdf.attrs = {"name": "my custom dataset"}
pdf.a.attrs = {"long_name": "Description about data", "nodata": -1, "units": "metre"}
pdf.to_parquet("file.parquet"){code}

Then, when loading in the data:
{code:python}
pdf = pandas.read_parquet("file.parquet")
pdf.attrs{code}
{"name": "my custom dataset"}
{code:java}
pdf.a.attrs{code}
{"long_name": "Description about data", "nodata": -1, "units": "metre"}



 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)