You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Mackenzie (JIRA)" <ji...@apache.org> on 2018/11/30 14:49:00 UTC

[jira] [Created] (ARROW-3915) [Python] Support partition columns when incrementally writing

Mackenzie created ARROW-3915:
--------------------------------

             Summary: [Python] Support partition columns when incrementally writing
                 Key: ARROW-3915
                 URL: https://issues.apache.org/jira/browse/ARROW-3915
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Python
    Affects Versions: 0.11.1
            Reporter: Mackenzie


Currently `partition_cols` support in pyarrow is implemented in: [https://github.com/apache/arrow/blob/69d207ff446c76f78fe27b960e7ebe89a607d992/python/pyarrow/parquet.py#L1205-L1235.]

However, there is no way to easily do column partitioning when writing datasets incrementally via `ParquetWriter`. It would be very helpful if the column partitioning logic was made more modular and re-used in `ParquetWriter`.

One option would be to support the `partition_cols` keyword argument in `ParquetWriter.write_table`. However, this would introduce the potential to have inconsistent partition columns in subsequent files. Perhaps the better approach would be to pass as a kwarg when constructing `ParquetWriter` and manage it as a property whose setter would throw an error if attempting to set while the writer is open.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)