You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/06/14 09:39:00 UTC
[PR] Apply filter in TopK [datafusion]
Dandandan opened a new pull request, #16408:
URL: https://github.com/apache/datafusion/pull/16408
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Apply filter early in TopK [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #16408:
URL: https://github.com/apache/datafusion/pull/16408#issuecomment-2972862984
Or you can just push to the main PR, I gave you write access to our fork :)
My one question is: how does this optimization play with filter pushdown? If a child plan accepted the filter as Exact should we then _not_ re-filter? A related question which you've alluded to before: if no child plan accepted the filter at all should we avoid updating it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Apply filter early in TopK [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #16408:
URL: https://github.com/apache/datafusion/pull/16408#issuecomment-2973088391
I applied the changes to https://github.com/apache/datafusion/pull/15770 let's continue there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Apply filter early in TopK [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #16408:
URL: https://github.com/apache/datafusion/pull/16408#issuecomment-2972888776
> Or you can just push to the main PR, I gave you write access to our fork :)
>
> My one question is: how does this optimization play with filter pushdown? If a child plan accepted the filter as Exact should we then _not_ re-filter? A related question which you've alluded to before: if no child plan accepted the filter at all should we avoid updating it?
I think avoiding running the filter twice for "exact cases" is optimal.
In practice, I am not sure if in this case it would add much overhead: converting to rows and comparing against / updating the heap / running the compaction logic will be the most expensive part by far.
It will be hard I think to show it being much slower somewhere.
Your second question: I think if we actually always use the filter for topk, I guess it isn't really wasteful. I think theoretically we should do it just before runing the topk instead of after (to avoid running it for the last iteration without using it). But also here I think it will be hard to show any benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Apply filter early in TopK [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed pull request #16408: Apply filter early in TopK
URL: https://github.com/apache/datafusion/pull/16408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Apply filter early in TopK [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #16408:
URL: https://github.com/apache/datafusion/pull/16408#issuecomment-2972860809
Can you make a PR on our fork and we merge this into the main PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org