You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@spark.apache.org by we...@apache.org on 2020/01/14 12:33:39 UTC

[spark] branch master updated (2688fae -> a2aa966)

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.


    from 2688fae  [SPARK-30498][ML][PYSPARK] Fix some ml parity issues between python and scala
     add a2aa966  [SPARK-29544][SQL] optimize skewed partition based on data size

No new revisions were added by this update.

Summary of changes:
 .../scala/org/apache/spark/MapOutputTracker.scala  |  62 +++--
 .../org/apache/spark/shuffle/ShuffleManager.scala  |  10 +-
 .../spark/shuffle/sort/SortShuffleManager.scala    |   9 +-
 .../org/apache/spark/sql/internal/SQLConf.scala    |  30 +++
 .../spark/sql/execution/ShuffledRowRDD.scala       |  23 +-
 .../execution/adaptive/AdaptiveSparkPlanExec.scala |   4 +
 .../execution/adaptive/LocalShuffledRowRDD.scala   |   6 +-
 .../adaptive/OptimizeLocalShuffleReader.scala      |   2 +-
 .../execution/adaptive/OptimizeSkewedJoin.scala    | 293 +++++++++++++++++++++
 .../adaptive/ReduceNumShufflePartitions.scala      | 125 +++++----
 .../execution/adaptive/SkewedShuffledRowRDD.scala  |  78 ++++++
 .../execution/exchange/ShuffleExchangeExec.scala   |  15 +-
 .../ReduceNumShufflePartitionsSuite.scala          |   5 +-
 .../adaptive/AdaptiveQueryExecSuite.scala          | 147 +++++++++++
 14 files changed, 703 insertions(+), 106 deletions(-)
 create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
 create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/SkewedShuffledRowRDD.scala


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org