[jira] [Commented] (ARROW-3947) [Python] query distinct values of a given partition from a ParquetDataset

["Ben Kietzman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230806#comment-17230806 ] 

Ben Kietzman commented on ARROW-3947:
-------------------------------------

[~XiUpsilon] when partition columns are selected on read they are materialized as dictionary arrays which include all unique values of the partition keys, so would it be sufficient to read from the ParquetDataset with only the partition columns selected?

> [Python] query distinct values of a given partition from a ParquetDataset
> -------------------------------------------------------------------------
>
>                 Key: ARROW-3947
>                 URL: https://issues.apache.org/jira/browse/ARROW-3947
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Python
>    Affects Versions: 0.10.0
>         Environment: MacOSX, Python 3.6, 
>            Reporter: Ji Xu
>            Priority: Minor
>              Labels: dataset-parquet-read, parquet
>
> Right now the values of a given partition from a `ParquetDataset` is buried inside `ParquetDataset.pieces`, a bit inconvenient for the user to dig out this information. A helper function/method to perform this task in `ParquetDataset` class would be very helpful for the users.
> A pure personal opinion on the name of this method: `ParquetDataset.select_distinct()` with partition_name as the positional arg, to resemble SQL `SELECT DISTINCT column FROM table`.
> I'm not sure how to contribute here on Jira, so I created this [GitHub Gist |https://gist.github.com/xujiboy/c3fcc47f720ed9adf2260c5d0ba8aed2]as an possible solution for this problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)