You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ypsah (via GitHub)" <gi...@apache.org> on 2025/05/26 20:27:01 UTC
[I] Improve performance of filter pre-processing [iceberg-python]
ypsah opened a new issue, #2037:
URL: https://github.com/apache/iceberg-python/issues/2037
### Feature Request / Improvement
Hi, apologies in advance for the rather broad scope.
I am using `pyiceberg` to store datasets partitioned in two dimensions and am using basic filtering to retrieve specific parts of the space, e.g. `(x = 0 AND y = 0) OR (x = 1 AND y = 1) OR (x = -1 and y = 2)`. Cardinality is relatively modest with a little over 200 partitions and depending on the dataset, data size can range from under a MiB to over 10GiB.
I have noticed that when I try to retrieve the whole dataset, but still supply the corresponding filter using only `AND`, `OR` and `=` operators, I pay a rather large overhead, and that overhead gets proportionally better if I factorize the filter using two `IN` operators, i.e. `x in (0, 1, 2, ...) AND y IN (-5, -4, -3, ...)`.
For the smallest datasets I manage, the difference is quite noticeable. To load the full dataset:
- without filters: 15s
- with `x IN (...) AND y IN (...)`: 45s
- with `(x = ... AND y = ...) OR (x = ... AND y = ...) OR ...`: 8m30s
I am planning on optimizing this on my side as much as possible, by factorizing my filters as much as possible (while still working around https://github.com/apache/iceberg-python/issues/1937 / https://github.com/apache/arrow/issues/46183), but I am wondering if we might be able to optimize things a little bit on `pyiceberg`'s side as well.
Most importantly, given my datasets are partitioned on `x` and `y`, little time is effectively spent _applying_ the filter, and it appears that most of the overhead is coming from pre-processing the expression objects themselves.
To highlight this, I produced a flamegraph for two runs, one using only the `=` comparison operator and another using `IN` operators instead (both filters covering the same partitions eventually), and using the `search` functionality of the resulting svg, I highlighted the samples that related to the `expression` package of `pyiceberg`:
<img width="1867" alt="py-spy record -o profile.svg -- python reproducer-equalto.py" src="https://github.com/user-attachments/assets/fabe479a-9b2a-49fb-963d-2eb123c298bd" />
<img width="934" alt="py-spy record -o profile.svg -- python reproducer-equalto.py" src="https://github.com/user-attachments/assets/b3e99af4-9e82-4055-92d2-89fde8a2cecc" />
Note how the flamegraph for the run that uses `=` spent 83.5% of its CPU time in code related to `expressions`, whereas the run that uses `IN` only spent 6.4% in the same parts of `pyiceberg`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-2962973689
Looking at the flame-graph, I think the difference between 0.8.x and 0.9.x is the addition of the residuals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-3684624879
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-3629618452
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "ypsah (via GitHub)" <gi...@apache.org>.
ypsah commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-2962121389
Unfortunately not. I get the same performance whether I use `0.9.1`, or `0.9.0`.
`0.8.1` does a bit better (4x improvement). Can try to bisect until I find the commit that introduces the regression.
Anyway, wether it's 8min or 2min, it feels pretty bad to have such a large overhead for a scan that otherwise takes 15sec unfiltered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-2962930037
@ypsah First of all, thanks for the elaborate issue. A bisect would be extremely helpful
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "jayceslesar (via GitHub)" <gi...@apache.org>.
jayceslesar commented on issue #2037:
URL: https://github.com/apache/iceberg-python/issues/2037#issuecomment-2954118828
Potentially related? https://github.com/apache/iceberg-python/pull/1830 if you run your test before this commit was merged (try 0.9.0) (You might need to increase the recursion limit on your system) do you see similar performance issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve performance of filter pre-processing [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2037: Improve performance of filter pre-processing
URL: https://github.com/apache/iceberg-python/issues/2037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org