You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jychen7 (via GitHub)" <gi...@apache.org> on 2023/04/14 03:15:23 UTC

[GitHub] [arrow-datafusion] jychen7 opened a new issue, #6000: Push down limit to SortPreservingMergeExec and SortPreservingMergeStream

jychen7 opened a new issue, #6000:
URL: https://github.com/apache/arrow-datafusion/issues/6000

   ### Is your feature request related to a problem or challenge?
   
   This is separated from https://github.com/apache/arrow-datafusion/issues/3516#issuecomment-1254006432.
   
   On a high level, #3516 implements the `green` path
   ```mermaid
   graph LR;
       SPME[SortPreservingMergeExec]-->spwn_handler-->SE[SortExec];
       SPME[SortPreservingMergeExec]-->streaming_merge-->SPMS[SortPreservingMergeStream];
       SE-->execute-->do_sort-->ExternalSort-->insert_batch-->sort_batch-->lexsort_to_indices;
       insert_batch-->spill-->in_mem_partial_sort-->get_sorted_iter-->lexsort_to_indices;
       ExternalSort-->sort-->streaming_merge;
       ExternalSort-->sort-->in_mem_partial_sort;
   
       style spwn_handler color:green
       style SE color:green
       style execute color:green
       style do_sort color:green
       style ExternalSort color:green
       style insert_batch color:green
       style sort_batch color:green
       style lexsort_to_indices color:green
       style sort color:green
       style in_mem_partial_sort color:green
       style get_sorted_iter color:green
       style spill color:green
   ```
   
   This should reduce memory usage of sort and hope this could help for https://github.com/apache/arrow-datafusion/issues/5969a 
   
   ### Describe the solution you'd like
   
   Push down the limit in `grey` path above 
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] Dandandan closed issue #6000: Push down limit to SortPreservingMergeExec and SortPreservingMergeStream

Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan closed issue #6000: Push down limit to SortPreservingMergeExec and SortPreservingMergeStream
URL: https://github.com/apache/arrow-datafusion/issues/6000


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb commented on issue #6000: Push down limit to SortPreservingMergeExec and SortPreservingMergeStream

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6000:
URL: https://github.com/apache/arrow-datafusion/issues/6000#issuecomment-1509046982

   FYI @tustvold 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] jychen7 commented on issue #6000: Push down limit to SortPreservingMergeExec and SortPreservingMergeStream

Posted by "jychen7 (via GitHub)" <gi...@apache.org>.
jychen7 commented on issue #6000:
URL: https://github.com/apache/arrow-datafusion/issues/6000#issuecomment-1509353600

   I try to implement the `fetch` in SortPreservingMergeStream. I read about Tournament_Tree, but not sure how to further improve it with TopK.
   
   It would be nice if have 2nd 👀 and see whether it is in a good direction.
   
   https://github.com/apache/arrow-datafusion/pull/6014/commits/b12e4546392e5dbb8a355b61373a3e1fc0c3f92d


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org