[jira] [Created] (ARROW-10248) [C++][Dataset] Dataset writing does not write schema metadata

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-10248:
---------------------------------------------

             Summary: [C++][Dataset] Dataset writing does not write schema metadata
                 Key: ARROW-10248
                 URL: https://issues.apache.org/jira/browse/ARROW-10248
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++
            Reporter: Joris Van den Bossche
             Fix For: 2.0.0


Not sure if this is related to the writing refactor that landed yesterday, but `write_dataset` does not preserve the schema metadata (eg used for pandas metadata):

{code}
In [20]: df = pd.DataFrame({'a': [1, 2, 3]})

In [21]: table = pa.Table.from_pandas(df)

In [22]: table.schema
Out[22]: 
a: int64
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 396

In [23]: ds.write_dataset(table, "test_write_dataset_pandas", format="parquet")

In [24]: pq.read_table("test_write_dataset_pandas/part-0.parquet").schema
Out[24]: 
a: int64
  -- field metadata --
  PARQUET:field_id: '1'
{code}

I tagged it for 2.0.0 for a moment in case it's possible today, but I didn't yet look into how easy it would be to fix.

cc [~bkietz]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)