[jira] [Created] (ARROW-17200) [Python, Parquet] support partitioning by Pandas DataFrame index

["Gregory Werbin (Jira)" <ji...@apache.org>]

Gregory Werbin created ARROW-17200:
--------------------------------------

             Summary: [Python, Parquet] support partitioning by Pandas DataFrame index
                 Key: ARROW-17200
                 URL: https://issues.apache.org/jira/browse/ARROW-17200
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Parquet, Python
            Reporter: Gregory Werbin


In a Pandas {{DataFrame}} with a multi-index, with a slowly-varying "outer" index level, one might want to partition by that index level when saving the data frame to Parquet format. This is currently not possible; you need to manually reset the index before writing, and re-add the index after reading. It would be very useful if you could supply the name of an index level to {{partition_cols}} instead of (or ideally in addition to) a data column name.

I originally posted this on the Pandas issue tracker ([https://github.com/pandas-dev/pandas/issues/47797]). Matthew Roeschke looked at the code and figured out that the partitioning functionality was implemented entirely in PyArrow, and that the change would need to happen within PyArrow itself.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)