[Python] - serializing pyarrow filter expression

[Ishan Anand <an...@outlook.com>]

Hi - I am working with a storage format meant to be PyArrow Dataset compatible. PyArrow datasets support specifying a filter, written using pyarrow.compute expressions, - [link](https://arrow.apache.org/docs/dev/python/generated/pyarrow.dataset.Expression.html).

Does the pyarrow API provide a mechanism to serialize compute expressions to a standard format like substrait? I want to analyze the filter expression, and push down some of its execution to the storage engine.

Note that casting the filter expression to a string and parsing it is an option, but things like the isin​ operator don't produce easy to parse strings.

```py
x = pc.field("colA")
z = (x > 3) & x.isin([10, 11])
str(z)

#  '((colA > 3) and is_in(colA, {value_set=int64:[\n  10,\n  11\n], skip_nulls=false}))'
```

Thank you,
Ishan

Re: [Python] - serializing pyarrow filter expression

[Alenka Frim via user <us...@arrow.apache.org>]

Hi Ishan,

I do not think there is an option to specify compute expression
with substrait at the moment.
There is a future plan to get it supported in the C++:
https://github.com/apache/arrow/issues/33985 and after we could bind it in
Python and we could use that functionality in PyArrow also.

Best,
Alenka

On Sun, Feb 26, 2023 at 9:26 AM Ishan Anand <an...@outlook.com> wrote:

> Hi - I am working with a storage format meant to be PyArrow Dataset
> compatible. PyArrow datasets support specifying a filter, written using
> pyarrow.compute expressions, - [link](
> https://arrow.apache.org/docs/dev/python/generated/pyarrow.dataset.Expression.html
> ).
>
> Does the pyarrow API provide a mechanism to serialize compute expressions
> to a standard format like substrait? I want to analyze the filter
> expression, and push down some of its execution to the storage engine.
>
> Note that casting the filter expression to a string and parsing it is an
> option, but things like the isin​ operator don't produce easy to parse
> strings.
>
> ```py
> x = pc.field("colA")
> z = (x > 3) & x.isin([10, 11])
> str(z)
>
> #  '((colA > 3) and is_in(colA, {value_set=int64:[\n  10,\n  11\n],
> skip_nulls=false}))'
> ```
>
> Thank you,
> Ishan
>
>