[jira] [Created] (SPARK-13626) SparkConf deprecation log messages are printed multiple times

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

Marcelo Vanzin created SPARK-13626:
--------------------------------------

             Summary: SparkConf deprecation log messages are printed multiple times
                 Key: SPARK-13626
                 URL: https://issues.apache.org/jira/browse/SPARK-13626
             Project: Spark
          Issue Type: Bug
          Components: Spark Core, SQL
    Affects Versions: 2.0.0
            Reporter: Marcelo Vanzin
            Priority: Minor


I noticed that if I have a deprecated config in my spark-defaults.conf, I'll see multiple warnings when running, for example, spark-shell. I collected the backtrace from when the messages are printed, and here's a few instances. The first one is the only one I expect to be printed.

{noformat}
java.lang.Exception:
        ...
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
        at org.apache.spark.repl.Main$.<init>(Main.scala:30)
{noformat}

The following ones are causing duplicate log messages and we should clean those up:

{noformat}
java.lang.Exception:
        at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
        ...
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
        at org.apache.spark.repl.Main$.createSparkContext(Main.scala:82)
{noformat}

{noformat}
java.lang.Exception:
        at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
        ...
        at org.apache.spark.SparkConf.setAll(SparkConf.scala:139)
        at org.apache.spark.SparkConf.clone(SparkConf.scala:358)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
        at org.apache.spark.repl.Main$.createSparkContext(Main.scala:98)
{noformat}

There are also a few more caused by the use of {{SparkConf.clone()}}.

{noformat}
java.lang.Exception:
        at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
        ...
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
        at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:48)
{noformat}

{noformat}
java.lang.Exception:
        at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
        ...
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
        at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
{noformat}

{noformat}
java.lang.Exception:
        at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
        ...
        at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
        at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
{noformat}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org