[jira] [Created] (SPARK-33465) RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-33465:
-----------------------------------

             Summary: RDD.takeOrdered should get rid of usage of reduce or use treeReduce instead
                 Key: SPARK-33465
                 URL: https://issues.apache.org/jira/browse/SPARK-33465
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.1.0
            Reporter: L. C. Hsieh
            Assignee: L. C. Hsieh


{{RDD.takeOrdered}} API sorts elements in each partition and puts the sorted elements into a `BoundedPriorityQueue`.  So actually in the result RDD each partition has only one priority queue. Then the API calls {{RDD.reduce}} API to reduce the elements. But as mentioned before the RDD has only one queue at each partition, it doesn't make sense to call reduce to reduce elements (here the element is queue).

We should either simplify {{RDD.reduce}} call in {{RDD.takeOrdered}} or replace it with {{treeReduce}} which can actually do partially reducing for this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org