Re: [PR] [SPARK-59748][CORE] Loser tree + small-k linear merge for UnsafeSorterSpillMerger [spark]

["david-mollitor-db (via GitHub)" <gi...@apache.org>]

david-mollitor-db commented on PR #58998:
URL: https://github.com/apache/spark/pull/58998#issuecomment-6195632336

   Hi @mridulm, you mentioned on #56804 that a loser tree gave a nontrivial improvement for the k-way merge, so I was hoping you might be able to take a look at this one when you have a moment. It replaces the `PriorityQueue` inside `UnsafeSorterSpillMerger` itself, so both the single-round merge and the final round of the bounded merge use it, with a linear scan for up to 8 runs and cached key prefixes. The description covers how it relates to #56804, and @vboo123 has reviewed it. Thank you!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org