You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2016/01/28 22:46:39 UTC
[jira] [Resolved] (SPARK-12926) SQLContext to display warning
message when non-sql configs are being set
[ https://issues.apache.org/jira/browse/SPARK-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-12926.
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 10849
[https://github.com/apache/spark/pull/10849]
> SQLContext to display warning message when non-sql configs are being set
> ------------------------------------------------------------------------
>
> Key: SPARK-12926
> URL: https://issues.apache.org/jira/browse/SPARK-12926
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Tejas Patil
> Priority: Trivial
> Fix For: 2.0.0
>
>
> Users unknowingly try to set core Spark configs in sqlContext but later realise that it didn't work.
> {color:red}
> scala> sqlContext.sql("SET spark.shuffle.memoryFraction=0.4")
> res3: org.apache.spark.sql.DataFrame = [key: string, value: string]
> scala> sqlContext.getConf.get("spark.shuffle.memoryFraction")
> java.util.NoSuchElementException: spark.shuffle.memoryFraction
> at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
> at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:193)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.SparkConf.get(SparkConf.scala:193)
> {color}
> We could do this:
> - for configs starting with "spark.", allow only if it starts with "spark.sql.". Otherwise disallow.
> - allow anything else.
> This will be a simple change in SqlConf :
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L621
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org