You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2025/03/12 21:44:02 UTC
[I] Add `ProgressiveEval` operator for optimize `SortPreservingMerge` [datafusion]
NGA-TRAN opened a new issue, #10488:
URL: https://github.com/apache/datafusion/issues/10488
### Is your feature request related to a problem or challenge?
Yes, this is a task of https://github.com/apache/datafusion/issues/10316. We will use the porting `ProgressiveEval` to optimize `SortPreservingMerge`
### Describe the solution you'd like
In InfluxDB IOx, when the inputs of `SortPreservingMerge` are all sorted on the sort key and their data do not overlap, we replace `SortPreservingMerge` with `ProgressiveEval` which:
1. Avoids starting all input streams at once
2. Avoids having to compare any keys (doesn't actually do a merge)
We wrote about using this operator here: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/
This ticket is to port `ProgressiveEval` from InfluxDB to DataFusion
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `ProgressiveEval` operator for optimize `SortPreservingMerge` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10488:
URL: https://github.com/apache/datafusion/issues/10488#issuecomment-2113297084
I think this ticket largely duplicates https://github.com/apache/datafusion/issues/10316 so let's use that instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `ProgressiveEval` operator for optimize `SortPreservingMerge` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10488: Add `ProgressiveEval` operator for optimize `SortPreservingMerge`
URL: https://github.com/apache/datafusion/issues/10488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Porting `ProgressiveEval` from InfluxDB IOx to optimize `SortPreservingMerge` [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on issue #10488:
URL: https://github.com/apache/datafusion/issues/10488#issuecomment-2108416444
I start working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `ProgressiveEval` operator for optimize `SortPreservingMerge` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10488: Add `ProgressiveEval` operator for optimize `SortPreservingMerge`
URL: https://github.com/apache/datafusion/issues/10488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `ProgressiveEval` for optimize `SortPreservingMerge` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10488:
URL: https://github.com/apache/datafusion/issues/10488#issuecomment-2113293884
I updated this ticket's description with some additional background and linked: https://www.influxdata.com/blog/making-recent-value-queries-hundreds-times-faster/
There is a PR for adding this operator here: https://github.com/apache/datafusion/pull/10490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add `ProgressiveEval` operator for optimize `SortPreservingMerge` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10488:
URL: https://github.com/apache/datafusion/issues/10488#issuecomment-2770297859
Duplicated by https://github.com/apache/datafusion/issues/15191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org