[GitHub] [spark] vicennial commented on a diff in pull request #40368: [SPARK-42748][CONNECT] Server-side Artifact Management

["vicennial (via GitHub)" <gi...@apache.org>]

vicennial commented on code in PR #40368:
URL: https://github.com/apache/spark/pull/40368#discussion_r1135233592


##########
core/src/main/scala/org/apache/spark/SparkContext.scala:
##########
@@ -386,6 +387,18 @@ class SparkContext(config: SparkConf) extends Logging {
     Utils.setLogLevel(Level.toLevel(upperCased))
   }
 
+  // Local storage for transferred artifacts through Spark Connect.
+  private val _sparkConnectArtifactDirectory: File = Utils.createTempDir("artifacts")
+
+  /**
+   * :: Experimental ::
+   * Returns the directory that stores artifacts transferred through Spark Connect.
+   *
+   * @since 3.4.0
+   */
+  @Experimental

Review Comment:
   This is primarily for access from the `connect` package but also carries a side benefit of allowing users to inspect the artifact directory. This is a temporary step for now (to pass information to the upper SQL layer) but it may be removed in the longer term (if we switch to passing information down to core from the SQL layer).
   
   No requirement for making it user-facing at the moment so any suggestions to limit scope (while still being able to access it from the `connect` package) would be welcome.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org