[GitHub] [spark] yaooqinn commented on a change in pull request #32144: [SPARK-35044][SQL] `SET propertyKey` shall also lookup `sparkSession.sharedState.hadoopConf` to display the effective default hive/hadoop configs

[GitBox <gi...@apache.org>]

yaooqinn commented on a change in pull request #32144:
URL: https://github.com/apache/spark/pull/32144#discussion_r612393515



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
##########
@@ -146,7 +146,9 @@ case class SetCommand(kv: Option[(String, Option[String])])
     // Queries a single property.
     case Some((key, None)) =>
       val runFunc = (sparkSession: SparkSession) => {
-        val value = sparkSession.conf.getOption(key).getOrElse("<undefined>")
+        val value = sparkSession.conf.getOption(key).getOrElse {
+          sparkSession.sharedState.hadoopConf.get(key, "<undefined>")

Review comment:
       > Can we make the PR title clearer?
   
   OK
   
   > BTW do we have a valid use case? e.g. a hadoop conf is not in `SQLConf` but in `sparkSession.sharedState.hadoopConf`.
   
   The pre-loaded ones from `core-site.xml, hive-site.xml` etc., will only stay in `sparkSession.sharedState.hadoopConf` or `sc. _hadoopConfiguation` not `SQLConf`. Some of them that related the Hive Metastore connection(never change it spark runtime), e.g. `hive.metastore.uris`, are clearly global static and unchangeable but displayable I guess. Some of the ones that might be related to, for example, the output codec/compression, preset in Hadoop/hive config files like `core-site.xml` shall bestill changeable from case to case, table to table, file to file, etc. So, it's meaningful to show the defaults for users to change based on that.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org