[GitHub] [spark] cloud-fan commented on a change in pull request #31868: [SPARK-34558][SQL][FOLLOWUP] Use final Hadoop conf to instantiate FileSystem in SharedState

[GitBox <gi...@apache.org>]

cloud-fan commented on a change in pull request #31868:
URL: https://github.com/apache/spark/pull/31868#discussion_r596176356



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
##########
@@ -53,24 +53,33 @@ private[sql] class SharedState(
   SharedState.setFsUrlStreamHandlerFactory(sparkContext.conf, sparkContext.hadoopConfiguration)
 
   private[sql] val (conf, hadoopConf) = {
-    // Load hive-site.xml into hadoopConf and determine the warehouse path which will be set into
-    // both spark conf and hadoop conf avoiding be affected by any SparkSession level options
-    val initialConfigsWithoutWarehouse = SharedState.resolveWarehousePath(
+    val warehousePath = SharedState.resolveWarehousePath(
       sparkContext.conf, sparkContext.hadoopConfiguration, initialConfigs)
 
     val confClone = sparkContext.conf.clone()
     val hadoopConfClone = new Configuration(sparkContext.hadoopConfiguration)
+    // Extract entries from `SparkConf` and put them in the Hadoop conf.
+    confClone.getAll.foreach { case (k, v) =>

Review comment:
       This is to follow `session.sessionState.newHadoopConf()`:
   1. The SQLConf contains all the configs from SparkConf, see https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala#L89
   2. The Hadoop conf contains all the configs from SQLConf, see https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala#L127




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org