You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/25 21:25:40 UTC
[GitHub] [arrow] babylone-star opened a new issue #11007: Can pyarrow filter parquet struct and list columns?
babylone-star opened a new issue #11007:
URL: https://github.com/apache/arrow/issues/11007
Take the following table stored via pyarrow into Apache Parquet:
| | id | regions |
| - | -- | -------------- |
| 0 | A | ['us', 'uk'] |
| 1 | B | ['uk', 'mx'] |
I'd like to filter the regions column via parquet when loading data. Something like this:
```python
import pyarrow.dataset as ds
dataset = ds.dataset("./example.parquet", format="parquet")
dataset.to_table(filter=ds.scalar('us').isin(ds.field('region')))
```
The expectation is that I would get back the first row, but not the second row.
This, however, does not work.The documentation does not have any useful information on how to do this kind of op. Is there any way of performing filters on more complex column types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] babylone-star edited a comment on issue #11007: Can pyarrow filter parquet struct and list columns?
Posted by GitBox <gi...@apache.org>.
babylone-star edited a comment on issue #11007:
URL: https://github.com/apache/arrow/issues/11007#issuecomment-906399172
Hey @westonpace thanks so much for the prompt response. Creating a kernel actually sounds like a really cool project. I may take you up on that offer when I have some spare time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #11007: Can pyarrow filter parquet struct and list columns?
Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #11007:
URL: https://github.com/apache/arrow/issues/11007#issuecomment-905911039
I think the short answer is that there is no way to handle this today but it is valid future work.
The `isin` expression maps to the `is_in` kernel which only tests if an element is in an array. What you are describing is probably a different kernel and it probably hasn't been written (I have a hard time keeping track of which kernels exist :) ). You'd be welcome to create such a kernel. I'd recommend by starting with a JIRA to create the kernel. It would be a "scalar kernel" similar to `match_substring` probably.
Once the kernel exists I think you would still need ARROW-11750 which will allow you to use any compute kernels as an "expression".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] babylone-star commented on issue #11007: Can pyarrow filter parquet struct and list columns?
Posted by GitBox <gi...@apache.org>.
babylone-star commented on issue #11007:
URL: https://github.com/apache/arrow/issues/11007#issuecomment-906399172
Hey @westonpace thanks so much for the prompt response. Creating a kernel actually sounds like a really cool project. I'll may take you up on that offer when I have some spare time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield closed issue #11007: Can pyarrow filter parquet struct and list columns?
Posted by GitBox <gi...@apache.org>.
emkornfield closed issue #11007:
URL: https://github.com/apache/arrow/issues/11007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org