Re: [I] Improve Table.sort_by performance [arrow]

["AlenkaF (via GitHub)" <gi...@apache.org>]

AlenkaF commented on issue #47832:
URL: https://github.com/apache/arrow/issues/47832#issuecomment-3410379140

   I do not understand Acero internals to be able to provide feedback, but, I did find this in the documentation: https://arrow.apache.org/docs/cpp/acero/user_guide.html#order-by-sink which states:
   
   > This node [order_by_sink] is a “pipeline breaker” and will fully materialize the dataset in memory. In the future, spillover mechanisms will be added which should alleviate this constraint.
   
   Might materializing the whole dataset limit the degree of parallelism?
   
   That being said, can you provide some more context? It is hard to comment on an enhancement proposal with so little information to start with. What is the size of the dataset when single thread is used? If the dataset is smaller, this still happens?Or is the dataset maybe too small to utilise multi-threading? I am not sure that "Table.sort_by always runs on a single thread" really holds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org