[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

[GitBox <gi...@apache.org>]

AngersZhuuuu commented on a change in pull request #31598:
URL: https://github.com/apache/spark/pull/31598#discussion_r579603890



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
##########
@@ -897,6 +898,36 @@ object SparkSession extends Logging {
       this
     }
 
+    // These submit configuration only effect when config before submit app.
+    private val SUBMIT_LAUNCHER_CONFIG =
+      Seq(SPARK_MASTER, DEPLOY_MODE, DRIVER_MEMORY, DRIVER_EXTRA_CLASSPATH,
+        DRIVER_DEFAULT_JAVA_OPTIONS, DRIVER_EXTRA_JAVA_OPTIONS, DRIVER_EXTRA_LIBRARY_PATH,
+        PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON, SPARKR_R_SHELL, CHILD_PROCESS_LOGGER_NAME,
+        CHILD_CONNECTION_TIMEOUT)
+
+    // These configuration can effect when SparkContext is not started.
+    private val EXECUTOR_LAUNCHER_CONFIG =
+      Seq(EXECUTOR_MEMORY, EXECUTOR_EXTRA_CLASSPATH, EXECUTOR_DEFAULT_JAVA_OPTIONS,
+        EXECUTOR_EXTRA_JAVA_OPTIONS, EXECUTOR_EXTRA_LIBRARY_PATH, EXECUTOR_CORES)
+
+    def checkAndSetConfig(key: String, value: String): Unit = {
+      if (SparkContext.getActive.isEmpty) {
+        if (SUBMIT_LAUNCHER_CONFIG.contains(key)) {
+          logWarning(s"Since spark has been started, configuration ${key} won't work" +
+            s" when set it here")

Review comment:
       > Can we have a directional guide instead of `xxx won't work when set it here`?
   
   Yea, I will make it more clear according to the doc you refer to.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org