You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2017/11/10 21:24:51 UTC
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19719
[SPARK-22487][SQL][followup] still keep spark.sql.hive.version
## What changes were proposed in this pull request?
a followup of https://github.com/apache/spark/pull/19712 , adds back the `spark.sql.hive.version`, so that if users try to read this config, they can still get a default value instead of null.
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark minor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19719.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19719
----
commit 7b767bfb010f7daeecbc8475c92bc1137e7e019a
Author: Wenchen Fan <we...@databricks.com>
Date: 2017-11-10T21:22:38Z
still keep spark.sql.hive.version
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83708/testReport)** for PR 19719 at commit [`0ce241d`](https://github.com/apache/spark/commit/0ce241da78c9a59fe255a76a6271313b5bc7d039).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150362312
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -521,7 +521,20 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.metastore.version") === Some("1.2.1"))
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ }
+ }
+
+ test("Checks Hive version via SET") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SET")
+
+ val conf = mutable.Map.empty[String, String]
+ while (resultSet.next()) {
+ conf += resultSet.getString(1) -> resultSet.getString(2)
+ }
+
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
--- End diff --
a default value will be returned, isn't it?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83732/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150366512
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
--- End diff --
it's ODBC. Actually it's something out of Spark's control, we must be careful and avoid behavior changes, like `conf.get("spark.sql.hive.version")` should still return `1.2.1` instead of null.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150363309
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -521,7 +521,20 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.metastore.version") === Some("1.2.1"))
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ }
+ }
+
+ test("Checks Hive version via SET") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SET")
+
+ val conf = mutable.Map.empty[String, String]
+ while (resultSet.next()) {
+ conf += resultSet.getString(1) -> resultSet.getString(2)
+ }
+
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
--- End diff --
Nope, I think. Try it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83708/testReport)** for PR 19719 at commit [`0ce241d`](https://github.com/apache/spark/commit/0ce241da78c9a59fe255a76a6271313b5bc7d039).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83713/testReport)** for PR 19719 at commit [`42d05a7`](https://github.com/apache/spark/commit/42d05a7f052d9a4f53030b35799c2ac479fb68f0).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19719
Thanks! Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83706/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150368620
--- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala ---
@@ -55,6 +55,7 @@ private[hive] object SparkSQLEnv extends Logging {
metadataHive.setOut(new PrintStream(System.out, true, "UTF-8"))
metadataHive.setInfo(new PrintStream(System.err, true, "UTF-8"))
metadataHive.setError(new PrintStream(System.err, true, "UTF-8"))
+ sparkSession.conf.set(HiveUtils.HIVE_EXECUTION_VERSION, HiveUtils.builtinHiveVersion)
--- End diff --
HiveUtils.builtinHiveVersion => HiveUtils.HIVE_EXECUTION_VERSION?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83723/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83705/testReport)** for PR 19719 at commit [`7b767bf`](https://github.com/apache/spark/commit/7b767bfb010f7daeecbc8475c92bc1137e7e019a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83713/testReport)** for PR 19719 at commit [`42d05a7`](https://github.com/apache/spark/commit/42d05a7f052d9a4f53030b35799c2ac479fb68f0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83723/testReport)** for PR 19719 at commit [`e37c1a8`](https://github.com/apache/spark/commit/e37c1a873719dbdda6ece75323921d09756e895c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83705/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83748/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19719
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19719
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83706/testReport)** for PR 19719 at commit [`dd1e344`](https://github.com/apache/spark/commit/dd1e3442b025052e3c72deaa049870de76c4dc0c).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150364544
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
+ val HIVE_EXECUTION_VERSION = buildConf("spark.sql.hive.version")
--- End diff --
doc may needed
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150368818
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -521,7 +521,20 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.metastore.version") === Some("1.2.1"))
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ }
+ }
+
+ test("Checks Hive version via SET") {
--- End diff --
i might be a session level conf
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19719
Add back the test cases?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19719
cc @gatorsmile @yaooqinn
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150366571
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
+ val HIVE_EXECUTION_VERSION = buildConf("spark.sql.hive.version")
--- End diff --
This config is a kind of a fake one, I think comments are enough for it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150402144
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,14 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A fake config which is only here for backward compatibility reasons. This config has no affect
+ // to Spark, just for reporting the builtin Hive version of Spark to existing applications that
+ // already reply on this config.
--- End diff --
`reply ` -> `rely`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83748/testReport)** for PR 19719 at commit [`b172ce9`](https://github.com/apache/spark/commit/b172ce9d899a1b589dd45f8f9795d3ba4d725696).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83733/testReport)** for PR 19719 at commit [`2088d6b`](https://github.com/apache/spark/commit/2088d6b0694f0d98b6f5c911c6e9f3f68590cd38).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19719
LGTM except a few comments
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83705/testReport)** for PR 19719 at commit [`7b767bf`](https://github.com/apache/spark/commit/7b767bfb010f7daeecbc8475c92bc1137e7e019a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83748/testReport)** for PR 19719 at commit [`b172ce9`](https://github.com/apache/spark/commit/b172ce9d899a1b589dd45f8f9795d3ba4d725696).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19719
Looking at `SQLConf.Deprecated.MAPRED_REDUCE_TASKS`, it needs to change many places, I don't think it worth for this config.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150358375
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -521,7 +521,20 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.metastore.version") === Some("1.2.1"))
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ }
+ }
+
+ test("Checks Hive version via SET") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SET")
+
+ val conf = mutable.Map.empty[String, String]
+ while (resultSet.next()) {
+ conf += resultSet.getString(1) -> resultSet.getString(2)
+ }
+
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
--- End diff --
If we do not set it explicitly, this will not be returned.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83732/testReport)** for PR 19719 at commit [`eb3137b`](https://github.com/apache/spark/commit/eb3137b8666b1ea57d1f9c03e37655b5f64a8461).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83733/testReport)** for PR 19719 at commit [`2088d6b`](https://github.com/apache/spark/commit/2088d6b0694f0d98b6f5c911c6e9f3f68590cd38).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19719
reverted `HiveThriftServer2Suites.scala` from #19712
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83712/testReport)** for PR 19719 at commit [`28fab02`](https://github.com/apache/spark/commit/28fab02358120f396f3272ca66ae3ac8b0534e4b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19719
Do something like the other deprecated conf `MAPRED_REDUCE_TASKS `?
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L980-L986
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83712/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150366200
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -521,7 +521,20 @@ class HiveThriftBinaryServerSuite extends HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.metastore.version") === Some("1.2.1"))
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ }
+ }
+
+ test("Checks Hive version via SET") {
+ withJdbcStatement() { statement =>
+ val resultSet = statement.executeQuery("SET")
+
+ val conf = mutable.Map.empty[String, String]
+ while (resultSet.next()) {
+ conf += resultSet.getString(1) -> resultSet.getString(2)
+ }
+
+ assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
--- End diff --
In Hive, "SET" returns all changed properties while "SET -v" returns all properties. In Spark, `SET` ueries all key-value pairs that are **set** in the SQLConf of the sparkSession.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83723/testReport)** for PR 19719 at commit [`e37c1a8`](https://github.com/apache/spark/commit/e37c1a873719dbdda6ece75323921d09756e895c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150364666
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
--- End diff --
i guess the jdbc driver do not depend on it. it used to only return `1.2.1`, but i think that this server may work fine with other hive versions
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83712/testReport)** for PR 19719 at commit [`28fab02`](https://github.com/apache/spark/commit/28fab02358120f396f3272ca66ae3ac8b0534e4b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83732/testReport)** for PR 19719 at commit [`eb3137b`](https://github.com/apache/spark/commit/eb3137b8666b1ea57d1f9c03e37655b5f64a8461).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19719
**[Test build #83706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83706/testReport)** for PR 19719 at commit [`dd1e344`](https://github.com/apache/spark/commit/dd1e3442b025052e3c72deaa049870de76c4dc0c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150402135
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,14 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A fake config which is only here for backward compatibility reasons. This config has no affect
--- End diff --
`affect` -> `effect`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83713/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83708/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150368744
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
+ val HIVE_EXECUTION_VERSION = buildConf("spark.sql.hive.version")
--- End diff --
or maybe more meaningful name
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19719
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83733/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19719: [SPARK-22487][SQL][followup] still keep spark.sql...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on a diff in the pull request:
https://github.com/apache/spark/pull/19719#discussion_r150368111
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala ---
@@ -66,6 +66,12 @@ private[spark] object HiveUtils extends Logging {
.stringConf
.createWithDefault(builtinHiveVersion)
+ // A deprecated config which is only used to provide a default value, in case some existing
+ // applications depend on this config, e.g. Spark SQL ODBC driver.
--- End diff --
if Thrift Server only works for 1.2.1，I guess we don’t need an isolated classload for its hive client like SparkSQLCLIDriver
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19719: [SPARK-22487][SQL][followup] still keep spark.sql.hive.v...
Posted by yaooqinn <gi...@git.apache.org>.
Github user yaooqinn commented on the issue:
https://github.com/apache/spark/pull/19719
make it an AlternativeConfig maybe better
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org