[GitHub] [spark] Ngone51 commented on a change in pull request #34018: [WIP][SPARK-36772] FinalizeShuffleMerge fails with an exception due to attempt id not matching

[GitBox <gi...@apache.org>]

Ngone51 commented on a change in pull request #34018:
URL: https://github.com/apache/spark/pull/34018#discussion_r710125319



##########
File path: core/src/main/scala/org/apache/spark/SparkEnv.scala
##########
@@ -355,6 +355,11 @@ object SparkEnv extends Logging {
       None
     }
 
+    // Set the application attemptId in the ExternalShuffleClient is applicable.
+    // If there is no attemptId assigned, set the attemptId to -1.
+    externalShuffleClient.foreach(
+      shuffleClient => shuffleClient.setAppAttemptId(conf.getInt(config.APP_ATTEMPT_ID.key, -1)))

Review comment:
       SparkEnv is created at L460:
   https://github.com/apache/spark/blob/adbea252db0f61ed3aaa45ea20b704bad6c47408/core/src/main/scala/org/apache/spark/SparkContext.scala#L460
   
   But application attemptId is set at L586 :
   https://github.com/apache/spark/blob/adbea252db0f61ed3aaa45ea20b704bad6c47408/core/src/main/scala/org/apache/spark/SparkContext.scala#L586
   
   So, isn't `conf.getInt(config.APP_ATTEMPT_ID.key, -1)` only return -1 here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org