[jira] [Updated] (ARROW-13369) [C++][python] performance of read_table using filters on a partitioned parquet file

["Weston Pace (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weston Pace updated ARROW-13369:
--------------------------------
    Summary: [C++][python] performance of read_table using filters on a partitioned parquet file  (was: performance of read_table using filters on a partitioned parquet file)

> [C++][python] performance of read_table using filters on a partitioned parquet file
> -----------------------------------------------------------------------------------
>
>                 Key: ARROW-13369
>                 URL: https://issues.apache.org/jira/browse/ARROW-13369
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>    Affects Versions: 4.0.0
>            Reporter: Richard Shadrach
>            Priority: Minor
>
> Reading a single partition of a parquet file via filters is significantly slower than reading the partition directly.
> {code:java}
> import pandas as pd
> size = 100_000
> df = pd.DataFrame({'a': [1, 2, 3] * size, 'b': [4, 5, 6] * size})
> df.to_parquet('test.parquet', partition_cols=['a'])
> %timeit pyarrow.parquet.read_table('test.parquet/a=1')
> %timeit pyarrow.parquet.read_table('test.parquet', filters=[('a', '=', 1)])
> {code}
> gives the timings
> {code:python}
> 2.57 ms ± 41.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
> 5.18 ms ± 148 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) {code}
> Likewise, changing size to 1_000_000 in the above code gives
> {code:python}
> 16.3 ms ± 269 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
> 32.7 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 10 loops each){code}
> Part of the docs for [read_table|https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html] states:
> > Partition keys embedded in a nested directory structure will be exploited to avoid loading files at all if they contain no matching rows.
> From this, I expected the performance to be roughly the same. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)