You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2021/12/23 12:11:00 UTC

[jira] [Created] (SPARK-37729) SparkSession.setLogLevel not working in Spark Shell

Hyukjin Kwon created SPARK-37729:
------------------------------------

             Summary: SparkSession.setLogLevel not working in Spark Shell
                 Key: SPARK-37729
                 URL: https://issues.apache.org/jira/browse/SPARK-37729
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.3.0
            Reporter: Hyukjin Kwon


In Spark 3.2:

{code}
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9

scala> spark.sparkContext.setLogLevel("WARN")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
21/12/23 21:08:18 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res5: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@7dafb9f9
{code}

In the current master:

{code}
scala> import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SparkSession

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137

scala> spark.sparkContext.setLogLevel("WARN")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137

scala> spark.sparkContext.setLogLevel("FATAL")

scala> SparkSession.builder.config("spark.abc", "abc").getOrCreate
res5: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@3e8a1137
{code}

Seems like it works when you set via {{setLogLevel}} initially but cannot be changed afterward.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org