You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "johnseekins (via GitHub)" <gi...@apache.org> on 2023/03/22 16:36:31 UTC

[GitHub] [arrow] johnseekins opened a new issue, #34688: Dataset -> Table -> Dataset loses partitions?

johnseekins opened a new issue, #34688:
URL: https://github.com/apache/arrow/issues/34688

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   This is a bit of a follow-on from https://github.com/apache/arrow/issues/34546
   
   So a rather long example:
   
   Initial table keys:
   ["key1", "key2", "key3", "partition1", "partition2"]
   
   Load to dataset with 
   ```
   partitions = ["partition1", "partition2"]
   data_set = ds.dataset(
       f"gs://bucket1/tables/parts",
       format="parquet",
       partitioning=partitions,
   )
   ```
   Then say
   ```
   >>> data_set.to_table().schema
   ["key1", "key2", "key3"]
   ```
   
   Clearly we're relying on the data layout to get the partitions. Which is fine and expected from how the docs describe it.
   
   Write that data out:
   ```
   table = polars.from_arrow(data_set.to_table())
   if table.height > 1:
       keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES]
       table = table.unique(subset=keys)
   table = table.to_arrow()
   ds.write_dataset(
       table,
       f"gs://{bucket}/{config['prefix']}/",
       format="parquet",
       file_options=file_options,
       partitioning=partitions,
       schema=table.schema,
       existing_data_behavior="overwrite_or_ignore",
   )
   ```
   
   We should have a file structure like:
   ```
   bucket1
    - tables
      - parts
        - partition1_value
          - partition2_value
            part-0.parquet
   ```
   
   Load that data:
   ```
   data_set = ds.dataset(
       f"gs://bucket1/tables/parts",
       format="parquet",
       partitioning=partitions,
   )
   ```
   
   Follow the same pattern:
   ```
   table = polars.from_arrow(data_set.to_table())
   if table.height > 1:
       keys = [k for k, v in table.schema.items() if v in POLARS_UNIQUE_TYPES]
       table = table.unique(subset=keys)
   table = table.to_arrow()
   ds.write_dataset(
       table,
       f"gs://{bucket}/{config['prefix']}/",
       format="parquet",
       file_options=file_options,
       partitioning=partitions,
       schema=table.schema,
       existing_data_behavior="overwrite_or_ignore",
   )
   ```
   Now we have:
   ```
   bucket1
    - tables
      - parts
        - part-0.parquet
        - partition1_value
          - partition2_value
            part-0.parquet
   ```
   
   And we've lost the partition columns. I _assume_ this is due to the conversion to polars and back.
   
   So my questions are:
   1. Is there a better way to do duplicate filtering in arrows than converting to polars and back?
   2. If we _are_ converting back and forth, how do we preserve the partitions from a dataset?
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] johnseekins closed issue #34688: Dataset -> Table -> Dataset loses partitions?

Posted by "johnseekins (via GitHub)" <gi...@apache.org>.
johnseekins closed issue #34688: Dataset -> Table -> Dataset loses partitions?
URL: https://github.com/apache/arrow/issues/34688


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] johnseekins commented on issue #34688: Dataset -> Table -> Dataset loses partitions?

Posted by "johnseekins (via GitHub)" <gi...@apache.org>.
johnseekins commented on issue #34688:
URL: https://github.com/apache/arrow/issues/34688#issuecomment-1489473730

   As far as I've been able to find, the correct answer is to basically not use datasets to try and load large amounts of data. And generally, using partitions with pyarrow seems to simply remove those columns from the actual data, which somewhat defeats the purpose of them. Closing this, but maybe some day we'll come back to it with renewed vigor!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org