You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2022/09/23 14:10:00 UTC
[jira] [Commented] (ARROW-3947) [Python] query distinct values of a given partition from a ParquetDataset
[ https://issues.apache.org/jira/browse/ARROW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608770#comment-17608770 ]
Joris Van den Bossche commented on ARROW-3947:
----------------------------------------------
Note that with the latest version of the dataset API, there is a {{partitioning}} attribute (instead of {{pieces}}), which can also be used to retrieve this information:
{code:python}
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.table({'a': range(9), 'part': ['a', 'b', 'c']*3})
pq.write_to_dataset(table, "test_partitioned", partition_cols=["part"])
dataset = pq.ParquetDataset("test_partitioned", use_legacy_dataset=False)
>>> dataset.partitioning
<pyarrow._dataset.HivePartitioning at 0x7fe760a1f580>
>>> dataset.partitioning.dictionaries
[<pyarrow.lib.StringArray object at 0x7fe740b6fdc0>
[
"a",
"b",
"c"
]]
{code}
Personally, I think this is sufficient API around this, but we should certainly do a better job documenting this.
> [Python] query distinct values of a given partition from a ParquetDataset
> -------------------------------------------------------------------------
>
> Key: ARROW-3947
> URL: https://issues.apache.org/jira/browse/ARROW-3947
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Affects Versions: 0.10.0
> Environment: MacOSX, Python 3.6,
> Reporter: Ji Xu
> Priority: Minor
> Labels: dataset-parquet-read, parquet
>
> Right now the values of a given partition from a `ParquetDataset` is buried inside `ParquetDataset.pieces`, a bit inconvenient for the user to dig out this information. A helper function/method to perform this task in `ParquetDataset` class would be very helpful for the users.
> A pure personal opinion on the name of this method: `ParquetDataset.select_distinct()` with partition_name as the positional arg, to resemble SQL `SELECT DISTINCT column FROM table`.
> I'm not sure how to contribute here on Jira, so I created this [GitHub Gist |https://gist.github.com/xujiboy/c3fcc47f720ed9adf2260c5d0ba8aed2]as an possible solution for this problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)