You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/03/14 22:28:33 UTC
[jira] [Resolved] (SPARK-13626) SparkConf deprecation log messages
are printed multiple times
[ https://issues.apache.org/jira/browse/SPARK-13626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-13626.
------------------------------------
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 2.0.0
> SparkConf deprecation log messages are printed multiple times
> -------------------------------------------------------------
>
> Key: SPARK-13626
> URL: https://issues.apache.org/jira/browse/SPARK-13626
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 2.0.0
>
>
> I noticed that if I have a deprecated config in my spark-defaults.conf, I'll see multiple warnings when running, for example, spark-shell. I collected the backtrace from when the messages are printed, and here's a few instances. The first one is the only one I expect to be printed.
> {noformat}
> java.lang.Exception:
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.repl.Main$.<init>(Main.scala:30)
> {noformat}
> The following ones are causing duplicate log messages and we should clean those up:
> {noformat}
> java.lang.Exception:
> at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.repl.Main$.createSparkContext(Main.scala:82)
> {noformat}
> {noformat}
> java.lang.Exception:
> at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.setAll(SparkConf.scala:139)
> at org.apache.spark.SparkConf.clone(SparkConf.scala:358)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:368)
> at org.apache.spark.repl.Main$.createSparkContext(Main.scala:98)
> {noformat}
> There are also a few more caused by the use of {{SparkConf.clone()}}.
> {noformat}
> java.lang.Exception:
> at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:59)
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:48)
> {noformat}
> {noformat}
> java.lang.Exception:
> at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
> at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
> {noformat}
> {noformat}
> java.lang.Exception:
> at org.apache.spark.SparkConf$$anonfun$logDeprecationWarning$2.apply(SparkConf.scala:682)
> ...
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:53)
> at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:93)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org