(spark) branch master updated: [SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors"

[wu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wuyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f077759460c9 [SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors"
f077759460c9 is described below

commit f077759460c9801ac46956b6c0e34a52d0a23ff4
Author: Bo Zhang <bo...@databricks.com>
AuthorDate: Fri Jun 21 10:37:50 2024 +0800

    [SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs for executors"
    
    ### What changes were proposed in this pull request?
    This change fixes test "error during accessing host local dirs for executors" in ShuffleBlockFetcherIteratorSuite, by setting up `blocksByAddress` correctly for the `ShuffleBlockFetcherIterator` used in the test.
    
    ### Why are the changes needed?
    The test is for host local blocks, but the `BlockManagerId` was not set up correctly, so `ShuffleBlockFetcherIterator` would treat those blocks as remote blocks instead.
    
    This change is to make the test valid to prevent further breaking changes.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Added an additional check in the test. Also tested locally and verified that the correct code path was reached.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #46989 from bozhang2820/spark-48631.
    
    Authored-by: Bo Zhang <bo...@databricks.com>
    Signed-off-by: Yi Wu <yi...@databricks.com>
---
 .../org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala   | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala b/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
index 02900e14b1f6..ca77d2c7b709 100644
--- a/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
@@ -392,9 +392,11 @@ class ShuffleBlockFetcherIteratorSuite extends SparkFunSuite with PrivateMethodT
 
     configureMockTransfer(Map())
     val iterator = createShuffleBlockIteratorWithDefaults(
-      Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1))
+      Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1)),
+      blockManager = Some(blockManager)
     )
     intercept[FetchFailedException] { iterator.next() }
+    verify(mockExternalBlockStoreClient, times(1)).getHostLocalDirs(any(), any(), any(), any())
   }
 
   test("Hit maxBytesInFlight limitation before maxBlocksInFlightPerAddress") {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org