You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "pgwhalen (via GitHub)" <gi...@apache.org> on 2026/03/30 13:46:49 UTC
[PR] docs: add custom table provider filter pushdown examples [datafusion]
pgwhalen opened a new pull request, #21250:
URL: https://github.com/apache/datafusion/pull/21250
- `pruning_predicate_filter_pushdown.rs` demonstrates filter pushdown that removes partitions from the scan using PruningPredicate
- `literal_guarantee_filter_pushdown.rs` demonstrates filter pushdown by directly pulling out literals using LiteralGuarantee and looking them up in a custom index
## Which issue does this PR close?
- Closes #21145.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21250: docs: add custom table provider filter pushdown examples
URL: https://github.com/apache/datafusion/pull/21250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen commented on PR #21250:
URL: https://github.com/apache/datafusion/pull/21250#issuecomment-4877461222
Looks like this got marked stale - obviously no rush to review, but bumping to keep it fresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen commented on PR #21250:
URL: https://github.com/apache/datafusion/pull/21250#issuecomment-4181098533
Thoughts for reviewers:
#### Does it make sense to include both examples?
I thought it was valuable to show both the pruning predicate method as well as literal guarantees directly, since my personal experience was that the pruning predicate was an unnecessary layer for me and I wanted to see the most direct way to get something useful from the filter expressions.
I'm also aware that [parquet_advanced_index.rs](https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/data_io/parquet_advanced_index.rs) has a lot of overlap with the pruning predicate example, so perhaps it makes sense to omit that. Though I think there's value in an example that doesn't show anything parquet specific and just focuses on generic partition elimination.
#### Should I reuse the code across examples (i.e. `User` struct, etc.)?
The other examples were pretty standalone so I followed that pattern, but obviously it's easy to switch.
#### Claude disclosure
This mostly Claude Code generated initially, but all reviewed/tweaked by me. Please let me know if it feels too LLM-ish still, I
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "pgwhalen (via GitHub)" <gi...@apache.org>.
pgwhalen commented on PR #21250:
URL: https://github.com/apache/datafusion/pull/21250#issuecomment-4364782387
@alamb might be a good candidate to review since he wrote the excellent `parquet_advanced_index.rs` example that helped me figure these out in the first place. @zhuqi-lucas also wrote a similar example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21250:
URL: https://github.com/apache/datafusion/pull/21250#issuecomment-4367130080
will try and find time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: add custom table provider filter pushdown examples [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21250:
URL: https://github.com/apache/datafusion/pull/21250#issuecomment-5519061211
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org