[jira] [Resolved] (ARROW-17483) [Python] Support for 'pa.compute.Expression' in filter argument to 'pa.read_table'

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-17483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche resolved ARROW-17483.
-------------------------------------------
    Fix Version/s: 10.0.0
       Resolution: Fixed

Issue resolved by pull request 14011
[https://github.com/apache/arrow/pull/14011]

> [Python] Support for 'pa.compute.Expression' in filter argument to 'pa.read_table'
> ----------------------------------------------------------------------------------
>
>                 Key: ARROW-17483
>                 URL: https://issues.apache.org/jira/browse/ARROW-17483
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: Patrik Kjærran
>            Assignee: Miles Granger
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 10.0.0
>
>          Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Currently, the _filters_ argument supports {{{}List{}}}[{{{}Tuple{}}}] or {{{}List{}}}[{{{}List{}}}[{{{}Tuple{}}}]] or None as its input types. I was suprised to see that Expressions were not supported, considering that filters are converted to expressions internally when using use_legacy_dataset=False.
> The check on [L150-L153|https://github.com/apache/arrow/blob/28cf3f9f769dda11ddfe52bd316c96aecb656522/python/pyarrow/parquet/core.py#L150-L153] short-circuits and succeeds when encountering an expression, but later fails on [L2343|https://github.com/apache/arrow/blob/28cf3f9f769dda11ddfe52bd316c96aecb656522/python/pyarrow/parquet/core.py#L2343] as the expression is evaluated as part of a boolean expression. 
> I think declaring filters using pa.compute.Expressions more pythonic and less error-prone,  and ill-formed filters will be detected much earlier than when using list-of-tuple-of-string equivalents.
> *Example:*
> {code:java}
> import pyarrow as pa
> import pyarrow.compute as pc
> import pyarrow.parquet as pq
> # Creating a dummy table
> table = pa.table({
>     'year': [2020, 2022, 2021, 2022, 2019, 2021],
>     'n_legs': [2, 2, 4, 4, 5, 100],
>     'animal': ["Flamingo", "Parrot", "Dog", "Horse", "Brittle stars", "Centipede"]
> })
> pq.write_to_dataset(table, root_path='dataset_name_2', partition_cols=['year'])
> # Reading using 'pyarrow.compute.Expression'
> pq.read_table('dataset_name_2', columns=["n_legs", "animal"], filters=pc.field("n_legs") < 4)
> # Reading using List[Tuple]
> pq.read_table('dataset_name_2', columns=["n_legs", "animal"], filters=[('n_legs', '<', 4)])  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)