You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zhzhan <gi...@git.apache.org> on 2015/01/15 20:57:44 UTC
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
GitHub user zhzhan opened a pull request:
https://github.com/apache/spark/pull/4064
[Spark-5111][SQL]HiveContext and Thriftserver cannot work in secure cluster beyond hadoop2.5
Hive0.13 cannot work with secure cluster in hadoop-2.5 and beyound. Due to "java.lang.NoSuchFieldError: SASL_PROPS" error. Need to backport some hive-0.14 fix into spark, since there is no effort to upgrade hive to 0.14 support in spark.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhzhan/spark spark5111
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4064.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4064
----
commit 3bf966c2f1bb913149a34176598a69041487cb88
Author: Zhan Zhang <zh...@gmail.com>
Date: 2014-08-08T17:47:18Z
test
commit fc56b25ff62964f59b96d2db13b5c357ae1c2f2b
Author: Zhan Zhang <zh...@gmail.com>
Date: 2015-01-07T21:01:45Z
squash all commits
commit c6b57402d19557105bc2bb95978b5815d7e95907
Author: Zhan Zhang <zh...@gmail.com>
Date: 2015-01-09T17:48:45Z
hive secure cluster fix
commit 456232c1ce29a7bff7f7d606764d5da00a478695
Author: Zhan Zhang <zh...@gmail.com>
Date: 2015-01-09T21:57:54Z
hive on secure cluster fix
commit 6532a342ba85be0300c169ce81f671da7ea5dcb1
Author: Zhan Zhang <zh...@gmail.com>
Date: 2015-01-15T19:53:36Z
rebase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87847743
[Test build #29420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29420/consoleFull) for PR 4064 at commit [`77871ca`](https://github.com/apache/spark/commit/77871cab8279d85da8b38f86d6c7d16e77053c05).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HadoopThriftAuthBridge23 extends HadoopThriftAuthBridge20S `
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87847777
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29420/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87141038
@liancheng By the way, if you think it can be merged to upstream, please let me know. Because SparkSQLCLIDriver also need to replace the method to make it work.
def main(args: Array[String]) {
+ HiveInstrumentationAgent.instrument
val oproc = new OptionsProcessor()
if (!oproc.process_stage1(args)) {
System.exit(1)
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87817047
[Test build #29420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29420/consoleFull) for PR 4064 at commit [`77871ca`](https://github.com/apache/spark/commit/77871cab8279d85da8b38f86d6c7d16e77053c05).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-87139435
@liancheng This is to port HIVE-7620 into spark to solve hadoop incompatibility issue. It will replace the getHadoopSaslProperties in org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S if necessary by checking the hadoop version.
https://issues.apache.org/jira/browse/HIVE-7620
I agree the code may not be fitting into the overall code base. But it is broken in hive-0.13 connecting to secure cluster hadoop-2.5 and beyond due to this issue. If hive-0.14 will be supported soon, this patch may not be critical.
By the way, hadoop-2.6 and beyond is becoming main stream in my point of view.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by zhzhan <gi...@git.apache.org>.
Github user zhzhan commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-70897525
@marmbrus Can you take a look at this and let you know if you have any concern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Spark-5111][SQL]HiveContext and Thriftserver ...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-86889339
@zhzhan Would you mind to give a summary of the code added in this PR? I feel kinda hesitant to have this... Mostly because Spark doesn't explicitly support Hadoop 2.5.0 and above versions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org