You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/08/15 12:26:03 UTC
[I] Parquet Pushdown Filters: Way to keep the I/O pattern the same when pushdown filters are enabled [datafusion]
alamb opened a new issue, #24393:
URL: https://github.com/apache/datafusion/issues/24393
### Is your feature request related to a problem or challenge?
- Related to https://github.com/apache/datafusion/issues/3463
- This is an idea for https://github.com/apache/datafusion/issues/20324
Namely, the idea is to avoid regressions when we turn on filter pushdown
We haven't been able to turn on predicate pushdown because it caused slowdowns. The slowdowns were caused by two reasons:
1. predicate pushdown changes the I/O patterns
2. Evaluating predicates is computationally slower in some cases (figuring out bitmasks, rows that passed, etc)
I think after the great work from @hhhizzz @haohuaijin and others upstream in arrow-rs/parquet we are close (if not already) at the point where it is computationally the same or faster
All that remains is the different I/O
The I/O pattern today is
1. Default: read all column data needed for both filtering and projections when reading a row group
2. Predicate Pushdown (`pushdown_filters`): sequentially fetch the columns required to evaluate filter expressions, and then after filtering is complete, fetches columns needed to evaluate the projections
@Dandandan / @adriangb also noted on https://github.com/apache/datafusion/pull/23492#issuecomment-5297913079 that we can't prune I/O for files without a page index (notably the clickbench dataset) anyways , so other than peak buffer needs, there is no benefit to changing the I/O to read columns sequentially in that case
### Describe the solution you'd like
I would like is some way to turn on filter pushdown but **keep the I/O pattern the same as today** (aka read all required data pages for a Row Gorup) up front in one request
That way, once we sort out any lingering CPU slowdowns, we can turn on predicate pushdown by default and leave the I/O pattern as a policy choice (as the I/O policy will likely different for different I/O systems -- e.g. NVMe vs ObjectStore.
### Describe alternatives you've considered
So I think we should separate out the notions of "filter evaluation mechanics" from the I/O pattern.
One idea is to add a parquet configuration setting like 'progressive_io' (defaults to false, the behavior today). When true it would fetch data progressively, as happens today when filter_pushdown is enabled
So we would have a feature matrix like
| filter_pushdown | progressive_io | description |
|--------|--------|--------|
| false | false | Same as todays behavior |
| true | false | filter pushdown, but still keep same I/O (my proposed new default) |
| false | true | N/A (just a decode, so no progressive filtering)
| true | true | progressive filter evaluation and I/O (what happens today when `filter_pushdown`) is enabled
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet Pushdown Filters: Way to keep the I/O pattern the same when pushdown filters are enabled [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24393:
URL: https://github.com/apache/datafusion/issues/24393#issuecomment-5302232683
Some related issues for context:
DataFusion (remaining CPU-side work):
- https://github.com/apache/datafusion/issues/20325
arrow-rs (where the progressive fetch / I/O mechanism lives):
- https://github.com/apache/arrow-rs/issues/8668 -- likely the closest existing upstream hook for fetching all pages up front with the push decoder
- https://github.com/apache/arrow-rs/issues/7983 -- the push decoder work that decouples I/O from decode, making an I/O policy split possible
arrow-rs work showing predicate evaluation is now computationally the same or faster:
- https://github.com/apache/arrow-rs/issues/7362
- https://github.com/apache/arrow-rs/issues/8203
Remaining upstream CPU-side work:
- https://github.com/apache/arrow-rs/issues/7456
- https://github.com/apache/arrow-rs/issues/9589
A prior concrete example of the I/O pattern problem described here:
- https://github.com/apache/arrow-rs/issues/8565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet Pushdown Filters: Way to keep the I/O pattern the same when pushdown filters are enabled [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #24393:
URL: https://github.com/apache/datafusion/issues/24393#issuecomment-5302215437
My plan is to create a prototype to see if this could be a viable solution and will report back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org