You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by scwf <gi...@git.apache.org> on 2014/09/23 03:00:45 UTC
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
GitHub user scwf opened a pull request:
https://github.com/apache/spark/pull/2499
[SPARK-3652] [SQL] upgrade spark sql hive version to 0.13.1
Now spark sql hive version is 0.12.0 and do not support 0.13.1 because of some api level changes in hive new version.
Since hive has backwards compatibility, this PR just upgrade the hive version to 0.13.1(compile this PR against 0.12.0 will get error), i think this is ok for users and we also do not need to support different version of hive .
Notes:
1. package cmd not changed, sbt/sbt -Phive assembly will get the assembly jar with hive 0.13.1
2. this PR use org.apache.hive since there is not a shaded one of org.spark-project.hive for 0.13.1
3. i regenerate golden answer since change of sql query result
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwf/spark hive-0.13.1-clean
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2499.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2499
----
commit 5d9de8ec6145d286ca05906d5e1dd1cfd9760e71
Author: scwf <wa...@huawei.com>
Date: 2014-09-21T15:45:00Z
update pom to org.apache.hive 0.13.1 version
commit c3aa95f9861a541df249f29ddb35b5ad9e6a4751
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T04:56:44Z
fix errors of hive/hive-thriftserver when update to org.apache.hive 0.13.1
commit 22f648655aa5941e53e65cbeadb097a32e0af8cf
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T06:00:55Z
fix StatisticsSuite error
commit f9fdc1ca944e14a986d910b7093da3ae4586cc68
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T06:38:01Z
loginFromKeyTab when set hive.server2.authentication
commit 2afcaa1e6f579b209ecc07d98a990520cdb81350
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T08:30:30Z
delete invalid set fs.default.name, this will lead to query error since SessionStat.start changed in hive0.13.1
commit a09fc4e37d54fda41c8cbf6afc6d577ece51ec55
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T08:42:28Z
fix Operation cancelled
commit 8b9309014e4e76560378a543fdddec51c874092c
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T09:09:08Z
regenerate golden answer
commit 9bee908fdc4ee947e2c96e8c0e9006f2023eb870
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T10:09:39Z
ignore stats_empty_partition
commit 0b15b748e94fd6afbc19cd4397cf9f74adf9064b
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T10:11:07Z
add logic for case VoidObjectInspector in method inspectorToDataType
commit eab2354187ce88c051ffc6c149847b08e532804b
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T10:39:51Z
reset TestHive in CachedTablesuite
commit 853632d71bdb16a6792776c951257524d728c8eb
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T14:34:52Z
fix Hivequerysuite
commit 6d5d0710eb2ab1c14208deb158c2f4b018ddbf33
Author: w00228970 <wa...@huawei.com>
Date: 2014-09-22T14:59:41Z
fix analyze MetastoreRelations
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57255001
ok, thanks. @liancheng, we can refer to the method of #2241 to provide a shim layer for hive-thriftserver to support 0.12 and 0.13, how about your idea?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57254210
It would be great to support the thrift server for both 0.12.0 and 0.13.0. Please discuss your design with @liancheng .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57289576
ok, i will have a try to use a shim layer to implement this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf closed the pull request at:
https://github.com/apache/spark/pull/2499
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57252766
The problem is metastore compatibility. Hive 0.13.0 cannot talk to a 0.12.0 metastore. For this reason we'll want to be able to support both in Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-56468081
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/142/consoleFull) for PR 2499 at commit [`6d5d071`](https://github.com/apache/spark/commit/6d5d0710eb2ab1c14208deb158c2f4b018ddbf33).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-56465504
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57252547
Hi , @marmbrus, thanks for your reply. I did not understand the break when upgrade to hive 0.13, in my understanding that sql syntax of hive 0.13 is compatible with 0.12, so for users there is no change. Maybe i am wrong understanding?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57280380
@scwf A shim layer seems reasonable if we can make clean abstractions. A major issue is that the original `HiveServer`/`HiveServer2` not designed to be extended by other applications, that's why we have to use reflection tricks to implement `HiveThriftServer2` and `SparkSQLCLIDriver`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57542448
Would it be okay to close this issue for now, and reopen it when you have a draft of the hive server? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57174261
@marmbrus, can you test this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57190760
Hi, thanks for working on this. Unfortunately, we can't just upgrade to Hive 0.13.0 as that would break users who are running 0.12.0. Instead, there is already a PR open to support both versions at the same time using a shim layer #2241. It would be great if you could make any comments on the approach there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-56468023
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/142/consoleFull) for PR 2499 at commit [`6d5d071`](https://github.com/apache/spark/commit/6d5d0710eb2ab1c14208deb158c2f4b018ddbf33).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3652] [SQL] upgrade spark sql hive vers...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/2499#issuecomment-57253952
Get it, i see in #2241 hive-thriftserver is not enabled, maybe i can make a new PR to cover hive-thriftserver based on this PR(already upgrade hive version of thriftserver to 0.13.1), how do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org