[GitHub] [arrow-datafusion] comphead commented on issue #6899: Optimize "per partition" top-k : `ROW_NUMBER < 5`

["comphead (via GitHub)" <gi...@apache.org>]

comphead commented on issue #6899:
URL: https://github.com/apache/arrow-datafusion/issues/6899#issuecomment-1631660067

   Spark does the similar way: it sorts and limits data per partition then sends the output to single partition where final sort/limit performed. Spark has the logic encapsulated in separate operator and looks like https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala#L311
   
   it also contains optimization like if tuples already ordered it will skip the excessive ordering, the same happens for projection 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org