[jira] [Resolved] (ARROW-13755) [Python] Allow usage of field_names in partitioning when saving datasets

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-13755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche resolved ARROW-13755.
-------------------------------------------
    Resolution: Fixed

Issue resolved by pull request 11008
[https://github.com/apache/arrow/pull/11008]

> [Python] Allow usage of field_names in partitioning when saving datasets
> ------------------------------------------------------------------------
>
>                 Key: ARROW-13755
>                 URL: https://issues.apache.org/jira/browse/ARROW-13755
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Alessandro Molina
>            Assignee: Alessandro Molina
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 6.0.0
>
>          Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> When loading back datasets, it's possible to quickly provide the name of the columns for which data was partitioned using
> {code}
> partitioning=pyarrow.dataset.partitioning(field_names=["year"])
> {code}
> this is convenient because it's easier and quicker than providing the whole schema, which can still be autodetected from the loaded data.
> On the other side, we don't support this when _saving_ data. If you provide {{field_names}} instead of the {{schema}} you will get a crash
> {code}
> pyarrow/dataset.py in _ensure_write_partitioning(scheme)
>     684     if not isinstance(scheme, Partitioning):
>     685         # TODO support passing field names, and get types from schema
> --> 686         raise ValueError("partitioning needs to be actual Partitioning object")
>     687     return scheme
>     688 
> {code}
> It would be convenient to allow to use {{field_names}} only even when saving as we can automatically detect the schema from the table itself that we are saving.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)