[spark] branch branch-3.2 updated: [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new 369781a  [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
369781a is described below

commit 369781a3311ab41acae3a747328f17629a528f4e
Author: Chandni Singh <si...@gmail.com>
AuthorDate: Mon Aug 2 23:35:32 2021 -0700

    [SPARK-36389][CORE][SHUFFLE] Revert the change that accepts negative mapId in ShuffleBlockId
    
    ### What changes were proposed in this pull request?
    With SPARK-32922, we added a change that ShuffleBlockId can have a negative mapId. This was to support push-based shuffle where -1 as mapId indicated a push-merged block. However with SPARK-32923, a different type of BlockId was introduced - ShuffleMergedId, but reverting the change to ShuffleBlockId was missed.
    
    ### Why are the changes needed?
    This reverts the changes to `ShuffleBlockId` which will never have a negative mapId.
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Modified the unit test to verify the newly added ShuffleMergedBlockId.
    
    Closes #33616 from otterc/SPARK-36389.
    
    Authored-by: Chandni Singh <si...@gmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 2712343a276a11b46f0771fe6a6d26ee1834a34f)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 core/src/main/scala/org/apache/spark/storage/BlockId.scala     |  2 +-
 .../src/test/scala/org/apache/spark/storage/BlockIdSuite.scala | 10 +++++-----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/storage/BlockId.scala b/core/src/main/scala/org/apache/spark/storage/BlockId.scala
index e450129..3d95c3b 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockId.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockId.scala
@@ -191,7 +191,7 @@ class UnrecognizedBlockId(name: String)
 @DeveloperApi
 object BlockId {
   val RDD = "rdd_([0-9]+)_([0-9]+)".r
-  val SHUFFLE = "shuffle_([0-9]+)_(-?[0-9]+)_([0-9]+)".r
+  val SHUFFLE = "shuffle_([0-9]+)_([0-9]+)_([0-9]+)".r
   val SHUFFLE_BATCH = "shuffle_([0-9]+)_([0-9]+)_([0-9]+)_([0-9]+)".r
   val SHUFFLE_DATA = "shuffle_([0-9]+)_([0-9]+)_([0-9]+).data".r
   val SHUFFLE_INDEX = "shuffle_([0-9]+)_([0-9]+)_([0-9]+).index".r
diff --git a/core/src/test/scala/org/apache/spark/storage/BlockIdSuite.scala b/core/src/test/scala/org/apache/spark/storage/BlockIdSuite.scala
index 2fb8fa4..4b7f1fa 100644
--- a/core/src/test/scala/org/apache/spark/storage/BlockIdSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/BlockIdSuite.scala
@@ -227,13 +227,13 @@ class BlockIdSuite extends SparkFunSuite {
   }
 
   test("merged shuffle id") {
-    val id = ShuffleBlockId(1, -1, 0)
-    assertSame(id, ShuffleBlockId(1, -1, 0))
-    assertDifferent(id, ShuffleBlockId(1, 1, 1))
-    assert(id.name === "shuffle_1_-1_0")
+    val id = ShuffleMergedBlockId(1, 2, 0)
+    assertSame(id, ShuffleMergedBlockId(1, 2, 0))
+    assertDifferent(id, ShuffleMergedBlockId(1, 3, 1))
+    assert(id.name === "shuffleMerged_1_2_0")
     assert(id.asRDDId === None)
     assert(id.shuffleId === 1)
-    assert(id.mapId === -1)
+    assert(id.shuffleMergeId === 2)
     assert(id.reduceId === 0)
     assertSame(id, BlockId(id.toString))
   }

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org