You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/12/23 12:41:00 UTC
[jira] [Commented] (SPARK-37727) Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate
[ https://issues.apache.org/jira/browse/SPARK-37727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464529#comment-17464529 ]
Apache Spark commented on SPARK-37727:
--------------------------------------
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/35001
> Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-37727
> URL: https://issues.apache.org/jira/browse/SPARK-37727
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> Currently, {{SparkSession.builder.getOrCreate()}} is too noisy even when duplicate configurations are set. And users cannot tell which configurations are to fix. See the example below:
> {code}
> ./bin/spark-shell --conf spark.abc=abc
> {code}
> {code}
> import org.apache.spark.sql.SparkSession
> spark.sparkContext.setLogLevel("DEBUG")
> SparkSession.builder.config("spark.abc", "abc").getOrCreate
> {code}
> {code}
> ...
> 21:12:40.601 [main] WARN org.apache.spark.sql.SparkSession - Using an existing SparkSession; some spark core configurations may not take effect.
> {code}
> This is strait forward when there are few configurations but it is difficult for users to figure out when there are too many configurations especially when these configurations are defined in property files like {{spark-default.conf}} that is sometimes maintained separately by system admins.
> See also https://github.com/apache/spark/pull/34757#discussion_r769248275
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org