[jira] [Assigned] (SPARK-18086) Regression: Hive variables no longer work in Spark 2.0

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-18086:
------------------------------------

    Assignee: Apache Spark

> Regression: Hive variables no longer work in Spark 2.0
> ------------------------------------------------------
>
>                 Key: SPARK-18086
>                 URL: https://issues.apache.org/jira/browse/SPARK-18086
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.0
>            Reporter: Ryan Blue
>            Assignee: Apache Spark
>
> The behavior of variables in the SQL shell has changed from 1.6 to 2.0. Specifically, --hivevar name=value and {{SET hivevar:name=value}} no longer work. Queries that worked correctly in 1.6 will either fail or produce unexpected results in 2.0 so I think this is a regression that should be addressed.
> Hive and Spark 1.6 work like this:
> 1. Command-line args --hiveconf and --hivevar can be used to set session properties. --hiveconf properties are added to the Hadoop Configuration.
> 2. {{SET}} adds a Hive Configuration property, {{SET hivevar:<name>=<value>}} adds a Hive var.
> 3. Hive vars can be substituted into queries by name, and Configuration properties can be substituted using {{hiveconf:name}}.
> In 2.0, hiveconf, sparkconf, and conf variable prefixes are all removed, then the value in SQLConf for the rest of the key is returned. SET adds properties to the session config and (according to [a comment|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala#L28]) the Hadoop configuration "during I/O".
> {code:title=Hive and Spark 1.6.1 behavior}
> [user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
> spark-sql> select "${hiveconf:test.conf}";
> 1
> spark-sql> select "${test.conf}";
> ${test.conf}
> spark-sql> select "${hivevar:test.var}";
> 2
> spark-sql> select "${test.var}";
> 2
> spark-sql> set test.set=3;
> SET test.set=3
> spark-sql> select "${test.set}"
> "${test.set}"
> spark-sql> select "${hivevar:test.set}"
> "${hivevar:test.set}"
> spark-sql> select "${hiveconf:test.set}"
> 3
> spark-sql> set hivevar:test.setvar=4;
> SET hivevar:test.setvar=4
> spark-sql> select "${hivevar:test.setvar}";
> 4
> spark-sql> select "${test.setvar}";
> 4
> {code}
> {code:title=Spark 2.0.0 behavior}
> [user@host:~]: spark-sql --hiveconf test.conf=1 --hivevar test.var=2
> spark-sql> select "${hiveconf:test.conf}";
> 1
> spark-sql> select "${test.conf}";
> 1
> spark-sql> select "${hivevar:test.var}";
> ${hivevar:test.var}
> spark-sql> select "${test.var}";
> ${test.var}
> spark-sql> set test.set=3;
> test.set        3
> spark-sql> select "${test.set}";
> 3
> spark-sql> set hivevar:test.setvar=4;
> hivevar:test.setvar      4
> spark-sql> select "${hivevar:test.setvar}";
> 4
> spark-sql> select "${test.setvar}";
> ${test.setvar}
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org