You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2023/01/10 14:25:16 UTC
[GitHub] [iceberg] martindurant commented on pull request #6555: Python: Expression to disjunctive normal form
martindurant commented on PR #6555:
URL: https://github.com/apache/iceberg/pull/6555#issuecomment-1377359789
You need this to get pyarrow to filter *within* row-groups? When used with dask, I am surprised you would need this, because ice would be in charge of filtering row groups, and dask would pass the filters directly to pyarrow. That's me guessing without looking at the detail of the code.
I must admit to having come up with the [original simple form](https://github.com/dask/fastparquet/commit/cba69795c0ee2c1ec9a4e276ec728f2b4ae6b2fc#diff-38b333399f04a93e200b35be164c2f66d8c1d99817ca5aff26b3fd01d5079fe9R194) of this, `("col", "==", 0)` and the AND form, but not the ANDOR extended form.
The docstring above is actually incorrect: fastparquet does also support row-wise filtering, but not via dask (and although you save on memory, it can be slower than loading everything and filtering after).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org