[GitHub] [spark] juliuszsompolski commented on a diff in pull request #43153: [SPARK-45360] Initialize spark session builder configuration from SPARK_REMOTE

["juliuszsompolski (via GitHub)" <gi...@apache.org>]

juliuszsompolski commented on code in PR #43153:
URL: https://github.com/apache/spark/pull/43153#discussion_r1339963445


##########
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##########
@@ -912,6 +912,11 @@ object SparkSession extends Logging {
     @deprecated(message = "Please use create() instead.", since = "3.5.0")
     def build(): SparkSession = create()
 
+    private def getClientConfiguration(): Configuration = {
+      // Use copy() to avoid modifying the state of the existing builder.
+      builder.copy().loadFromEnvironment().configuration

Review Comment:
   in this case, SPARK_REMOTE will take precedence over what was explicitly set in the builder, right?
   so having in env
   ```
   SPARK_REMOTE="sc://foo:123"
   ```
   and calling
   ```
   SparkSession.builder.remote("sc://bar:234").create()
   ```
   then the SPARK_REMOTE will take precedence? Documentation does not specify precedence, but it feels like setting `remote` explicitly should take precedence over env var (and it should also be specified in the documentation)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org