[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42731: [SPARK-45014][CONNECT] Clean up fileserver when cleaning up files, jars and archives in SparkContext

["HyukjinKwon (via GitHub)" <gi...@apache.org>]

HyukjinKwon commented on code in PR #42731:
URL: https://github.com/apache/spark/pull/42731#discussion_r1311036686


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/artifact/SparkConnectArtifactManager.scala:
##########
@@ -208,12 +209,38 @@ class SparkConnectArtifactManager(sessionHolder: SessionHolder) extends Logging
         s"sessionId: ${sessionHolder.sessionId}")
 
     // Clean up added files
-    sessionHolder.session.sparkContext.addedFiles.remove(state.uuid)
-    sessionHolder.session.sparkContext.addedArchives.remove(state.uuid)
-    sessionHolder.session.sparkContext.addedJars.remove(state.uuid)
+    val fileserver = SparkEnv.get.rpcEnv.fileServer
+    val sparkContext = sessionHolder.session.sparkContext
+
+    sparkContext.synchronized {
+      val allAddedFiles = sparkContext.allAddedFiles.keys
+      val allAddedArchives = sparkContext.allAddedArchives.keys
+      val allAddedJars = sparkContext.allAddedJars.keys
+
+      val removedFiles = sparkContext.addedFiles.remove(state.uuid)
+      val removedArchives = sparkContext.addedArchives.remove(state.uuid)
+      val removedJars = sparkContext.addedJars.remove(state.uuid)
+
+      // In case there are duplicate files added by other sessions.

Review Comment:
   Actually yeah. Let's isolate them. Let me push some fixes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org