You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ala <gi...@git.apache.org> on 2018/02/23 14:45:25 UTC

[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

GitHub user ala opened a pull request:

    https://github.com/apache/spark/pull/20664

    [SPARK-23496][CORE] Locality of coalesced partitions can be severely skewed by the order of input partitions

    ## What changes were proposed in this pull request?
    
    The algorithm in `DefaultPartitionCoalescer.setupGroups` is responsible for picking preferred locations for coalesced partitions. It analyzes the preferred locations of input partitions. It starts by trying to create one partition for each unique location in the input. However, if the the requested number of coalesced partitions is higher that the number of unique locations, it has to pick duplicate locations.
    
    Previously, the duplicate locations would be picked by iterating over the input partitions in order, and copying their preferred locations to coalesced partitions. If the input partitions were clustered by location, this could result in severe skew.
    
    With the fix, instead of iterating over the list of input partitions in order, we pick them at random. It's not perfectly balanced, but it's much better.
    
    ## How was this patch tested?
    
    Unit test reproducing the behavior was added.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ala/spark SPARK-23496

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20664
    
----
commit 6d67dfc1d4c012492b97873beaac5a7cbfd6f55a
Author: Ala Luszczak <al...@...>
Date:   2018-02-23T14:37:19Z

    Fix SPARK-23496.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1143/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87771/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87827/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1100/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87715 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87715/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87697/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87632/testReport)** for PR 20664 at commit [`6d67dfc`](https://github.com/apache/spark/commit/6d67dfc1d4c012492b97873beaac5a7cbfd6f55a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87697/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87715/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87954/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87632/testReport)** for PR 20664 at commit [`6d67dfc`](https://github.com/apache/spark/commit/6d67dfc1d4c012492b97873beaac5a7cbfd6f55a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170279656
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    --- End diff --
    
    I see, thanks, sorry, I missed it


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170612006
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    +    val numInputPartitions = 100
    +    val numCoalescedPartitions = 50
    +    val locations = Array("locA", "locB")
    +
    +    val inputRDD = sc.makeRDD(Range(0, numInputPartitions).toArray[Int], numInputPartitions)
    +    assert(inputRDD.getNumPartitions == numInputPartitions)
    +
    +    val locationPrefRDD = new LocationPrefRDD(inputRDD, { (p: Partition) =>
    +      if (p.index < numCoalescedPartitions) {
    +        Seq(locations(0))
    +      } else {
    +        Seq(locations(1))
    +      }
    +    })
    +    val coalescedRDD = new CoalescedRDD(locationPrefRDD, numCoalescedPartitions)
    +
    +    val numPartsPerLocation = coalescedRDD
    +      .getPartitions
    +      .map(coalescedRDD.getPreferredLocations(_).head)
    +      .groupBy(identity)
    +      .mapValues(_.size)
    +
    +    // Without the fix these would be:
    --- End diff --
    
    Done.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87827/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87954/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1183/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87715/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170456391
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    +    val numInputPartitions = 100
    +    val numCoalescedPartitions = 50
    +    val locations = Array("locA", "locB")
    +
    +    val inputRDD = sc.makeRDD(Range(0, numInputPartitions).toArray[Int], numInputPartitions)
    +    assert(inputRDD.getNumPartitions == numInputPartitions)
    +
    +    val locationPrefRDD = new LocationPrefRDD(inputRDD, { (p: Partition) =>
    +      if (p.index < numCoalescedPartitions) {
    +        Seq(locations(0))
    +      } else {
    +        Seq(locations(1))
    +      }
    +    })
    +    val coalescedRDD = new CoalescedRDD(locationPrefRDD, numCoalescedPartitions)
    +
    +    val numPartsPerLocation = coalescedRDD
    +      .getPartitions
    +      .map(coalescedRDD.getPreferredLocations(_).head)
    +      .groupBy(identity)
    +      .mapValues(_.size)
    +
    +    // Without the fix these would be:
    +    //  numPartsPerLocation(locations(0)) == numCoalescedPartitions - 1
    +    //  numPartsPerLocation(locations(1)) == 1
    +    assert(numPartsPerLocation(locations(0)) > 0.4 * numCoalescedPartitions)
    --- End diff --
    
    nvm, the result is deterministic.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1019/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170277224
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    --- End diff --
    
    The test is in fact deterministic. The seed is already fixed here: https://github.com/apache/spark/blob/049f243c59737699fee54fdc9d65cbd7c788032a/core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala#L163


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by mgaido91 <gi...@git.apache.org>.
Github user mgaido91 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170271696
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    --- End diff --
    
    this test looks to me as a good candidate for flakiness, since we are are picking random numbers. Can we set the seed in order to avoid this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1270/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1059/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87697/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87827/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87672/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170456039
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    +    val numInputPartitions = 100
    +    val numCoalescedPartitions = 50
    +    val locations = Array("locA", "locB")
    +
    +    val inputRDD = sc.makeRDD(Range(0, numInputPartitions).toArray[Int], numInputPartitions)
    +    assert(inputRDD.getNumPartitions == numInputPartitions)
    +
    +    val locationPrefRDD = new LocationPrefRDD(inputRDD, { (p: Partition) =>
    +      if (p.index < numCoalescedPartitions) {
    +        Seq(locations(0))
    +      } else {
    +        Seq(locations(1))
    +      }
    +    })
    +    val coalescedRDD = new CoalescedRDD(locationPrefRDD, numCoalescedPartitions)
    +
    +    val numPartsPerLocation = coalescedRDD
    +      .getPartitions
    +      .map(coalescedRDD.getPreferredLocations(_).head)
    +      .groupBy(identity)
    +      .mapValues(_.size)
    +
    +    // Without the fix these would be:
    +    //  numPartsPerLocation(locations(0)) == numCoalescedPartitions - 1
    +    //  numPartsPerLocation(locations(1)) == 1
    +    assert(numPartsPerLocation(locations(0)) > 0.4 * numCoalescedPartitions)
    --- End diff --
    
    How confident are we on the assert condition to be true? How is the fraction `0.4` chosen?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170456095
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    +    val numInputPartitions = 100
    +    val numCoalescedPartitions = 50
    +    val locations = Array("locA", "locB")
    +
    +    val inputRDD = sc.makeRDD(Range(0, numInputPartitions).toArray[Int], numInputPartitions)
    +    assert(inputRDD.getNumPartitions == numInputPartitions)
    +
    +    val locationPrefRDD = new LocationPrefRDD(inputRDD, { (p: Partition) =>
    +      if (p.index < numCoalescedPartitions) {
    +        Seq(locations(0))
    +      } else {
    +        Seq(locations(1))
    +      }
    +    })
    +    val coalescedRDD = new CoalescedRDD(locationPrefRDD, numCoalescedPartitions)
    +
    +    val numPartsPerLocation = coalescedRDD
    +      .getPartitions
    +      .map(coalescedRDD.getPreferredLocations(_).head)
    +      .groupBy(identity)
    +      .mapValues(_.size)
    +
    +    // Without the fix these would be:
    --- End diff --
    
    Normally we don't write the comment this way, maybe just saying we want to ensure the location preferences distribute evenly.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170455929
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala ---
    @@ -266,17 +266,15 @@ private class DefaultPartitionCoalescer(val balanceSlack: Double = 0.10)
             numCreated += 1
           }
         }
    -    tries = 0
         // if we don't have enough partition groups, create duplicates
         while (numCreated < targetLen) {
    -      val (nxt_replica, nxt_part) = partitionLocs.partsWithLocs(tries)
    -      tries += 1
    +      val (nxt_replica, nxt_part) = partitionLocs.partsWithLocs(
    --- End diff --
    
    Perhaps add comment to explain the purpose of this change here?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87672/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87672/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Thanks for the comments.
    
    I don't think the users should be impacted by changing execution time. If the parameters of the job are constant, then the partition allocation should also be deterministic, since the seed is fixed in `CoalescedRDD.scala`. There was already a degree of randomization in `DefaultPartitionCoalescer.pickBin()` which could lead to some fluctuation, so it's not a big difference.
    
    TBH, I'm just trying to merge upstream a fix we've implemented for the client. I agree much more could be done to improve coalesce, and if someone would be interested in looking into it, I'm all for it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87954/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    @hvanhovell 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87771/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1085/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merging to master. Thanks!
    
    @mgaido91 if you feel this should be different, feel free to open a follow-up.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    **[Test build #87771 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87771/testReport)** for PR 20664 at commit [`0512736`](https://github.com/apache/spark/commit/051273651cd65b9eca568b37c79b50342a7f69c2).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20664


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87632/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20664: [SPARK-23496][CORE] Locality of coalesced partiti...

Posted by ala <gi...@git.apache.org>.
Github user ala commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20664#discussion_r170611946
  
    --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala ---
    @@ -1129,6 +1129,36 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {
         }.collect()
       }
     
    +  test("SPARK-23496: order of input partitions can result in severe skew in coalesce") {
    +    val numInputPartitions = 100
    +    val numCoalescedPartitions = 50
    +    val locations = Array("locA", "locB")
    +
    +    val inputRDD = sc.makeRDD(Range(0, numInputPartitions).toArray[Int], numInputPartitions)
    +    assert(inputRDD.getNumPartitions == numInputPartitions)
    +
    +    val locationPrefRDD = new LocationPrefRDD(inputRDD, { (p: Partition) =>
    +      if (p.index < numCoalescedPartitions) {
    +        Seq(locations(0))
    +      } else {
    +        Seq(locations(1))
    +      }
    +    })
    +    val coalescedRDD = new CoalescedRDD(locationPrefRDD, numCoalescedPartitions)
    +
    +    val numPartsPerLocation = coalescedRDD
    +      .getPartitions
    +      .map(coalescedRDD.getPreferredLocations(_).head)
    +      .groupBy(identity)
    +      .mapValues(_.size)
    +
    +    // Without the fix these would be:
    +    //  numPartsPerLocation(locations(0)) == numCoalescedPartitions - 1
    +    //  numPartsPerLocation(locations(1)) == 1
    +    assert(numPartsPerLocation(locations(0)) > 0.4 * numCoalescedPartitions)
    --- End diff --
    
    Added comment about flakiness & fixed seed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20664: [SPARK-23496][CORE] Locality of coalesced partitions can...

Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/20664
  
    This LGTM overall, just some nits.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org