[jira] [Updated] (ARROW-10248) [C++][Dataset] Dataset writing does not write schema metadata

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-10248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated ARROW-10248:
-----------------------------------
    Labels: pull-request-available  (was: )

> [C++][Dataset] Dataset writing does not write schema metadata
> -------------------------------------------------------------
>
>                 Key: ARROW-10248
>                 URL: https://issues.apache.org/jira/browse/ARROW-10248
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++
>            Reporter: Joris Van den Bossche
>            Assignee: Ben Kietzman
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.0.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Not sure if this is related to the writing refactor that landed yesterday, but `write_dataset` does not preserve the schema metadata (eg used for pandas metadata):
> {code}
> In [20]: df = pd.DataFrame({'a': [1, 2, 3]})
> In [21]: table = pa.Table.from_pandas(df)
> In [22]: table.schema
> Out[22]: 
> a: int64
> -- schema metadata --
> pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 396
> In [23]: ds.write_dataset(table, "test_write_dataset_pandas", format="parquet")
> In [24]: pq.read_table("test_write_dataset_pandas/part-0.parquet").schema
> Out[24]: 
> a: int64
>   -- field metadata --
>   PARQUET:field_id: '1'
> {code}
> I tagged it for 2.0.0 for a moment in case it's possible today, but I didn't yet look into how easy it would be to fix.
> cc [~bkietz]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)