You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xudong963 (via GitHub)" <gi...@apache.org> on 2025/10/15 09:32:53 UTC
[I] Add different configs for topk/join dynamic filter [datafusion]
xudong963 opened a new issue, #18071:
URL: https://github.com/apache/datafusion/issues/18071
Currently, we use the `enable_dynamic_filter_pushdown` config to control topk and join dynamic filter iiuc.
I think it'll be better to give users more flexibility to choose which dynamic filter. Such as, they may only need topk dynamic filter, or there are some bugs in join dynamic filter.
Proposal:
1. Add two new configuration options to replace the single `enable_dynamic_filter_pushdown`:
- `enable_topk_dynamic_filter_pushdown` - for TopKdynamic filters
- `enable_join_dynamic_filter_pushdown` - for Join dynamic filters
2. Keep the old config for backward compatibility, make it control both new configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add different configs for topk/join dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #18071:
URL: https://github.com/apache/datafusion/issues/18071#issuecomment-3409035425
Makes sense, happy to approve a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add different configs for topk/join dynamic filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #18071:
URL: https://github.com/apache/datafusion/issues/18071#issuecomment-3405878115
I have no problem with this, but I'd be interested to understand why a user might want to turn off the feature. Yes there are many cases (distributed, some custom operator that doesn't implement the pushdown methods, etc.) where pushdown doesn't work, but if it does work why do users want to turn it off?
In my mind the configs were more of a nice thing for us to be able to debug, fuzz test, etc. than something users would be configuring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add different configs for topk/join dynamic filter [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #18071: Add different configs for topk/join dynamic filter
URL: https://github.com/apache/datafusion/issues/18071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add different configs for topk/join dynamic filter [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #18071:
URL: https://github.com/apache/datafusion/issues/18071#issuecomment-3408963176
I'm planning to use the dynamic filter in production. I hope we can disable one of them in case one has an unexpected bug, but we can still use another one.
Yes, separated configs are also good for debugging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org