[jira] [Created] (SPARK-3409) Avoid pulling in Exchange operator itself in Exchange's closures

["Reynold Xin (JIRA)" <ji...@apache.org>]

Reynold Xin created SPARK-3409:
----------------------------------

             Summary: Avoid pulling in Exchange operator itself in Exchange's closures
                 Key: SPARK-3409
                 URL: https://issues.apache.org/jira/browse/SPARK-3409
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Reynold Xin
            Assignee: Reynold Xin


{code}
        val rdd = child.execute().mapPartitions { iter =>
          if (sortBasedShuffleOn) {
            iter.map(r => (null, r.copy()))
          } else {
            val mutablePair = new MutablePair[Null, Row]()
            iter.map(r => mutablePair.update(null, r))
          }
        }
{code}

The above snippet from Exchange references sortBasedShuffleOn within a closure, which requires pulling in the entire Exchange object in the closure. 

This is a tiny teeny optimization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org