[jira] [Commented] (SPARK-7961) Redesign SQLConf for better error message reporting

["Reynold Xin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565803#comment-14565803 ] 

Reynold Xin commented on SPARK-7961:
------------------------------------

One other thing that'd be nice as part of this redesign is to be able to support some sort of list all config options, their values, and the documentation for them in a sql command. We should mark certain config options as internal so they don't get displayed.


> Redesign SQLConf for better error message reporting
> ---------------------------------------------------
>
>                 Key: SPARK-7961
>                 URL: https://issues.apache.org/jira/browse/SPARK-7961
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>            Priority: Blocker
>
> Right now, we don't validate config values and as a result will throw exceptions when queries or DataFrame operations are run.
> Imagine if one user sets config variable "spark.sql.retainGroupColumns" (requires "true", "false") to "hello". The set action itself will complete fine. When another user runs a query, it will throw the following exception:
> {code}
> java.lang.IllegalArgumentException: For input string: "hello"
>     at scala.collection.immutable.StringLike$class.parseBoolean(StringLike.scala:238)
>     at scala.collection.immutable.StringLike$class.toBoolean(StringLike.scala:226)
>     at scala.collection.immutable.StringOps.toBoolean(StringOps.scala:31)
>     at org.apache.spark.sql.SQLConf.dataFrameRetainGroupColumns(SQLConf.scala:265)
>     at org.apache.spark.sql.GroupedData.toDF(GroupedData.scala:74)
>     at org.apache.spark.sql.GroupedData.agg(GroupedData.scala:227)
> {code}
> This is highly confusing. We should redesign SQLConf to validate data input at set time (during setConf call).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org