You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2017/04/10 01:39:18 UTC
[GitHub] zeppelin pull request #2237: ZEPPELIN-2377. Hive Support can not be enabled ...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2237
ZEPPELIN-2377. Hive Support can not be enabled in spark master
### What is this PR for?
The root cause is that `org.apache.spark.sql.hive.HiveSessionState` is removed in spark master. I change it to `org.apache.spark.sql.hive.execution.InsertIntoHiveTable` which is existed early in spark 1.0.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2377
### How should this be tested?
Verify it manually in spark master, spark 2.1.0 and spark 1.6.2
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2377
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2237.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2237
----
commit 81cd6b171018dc1cb05c7d9a10a2d4d5ade40cca
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-04-09T07:46:32Z
ZEPPELIN-2377. Hive Support can not be enabled in spark master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2237
For
hiveClassesArePresent
I mean either call hiveClassesArePresent (it's private yes) or do what it does?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2237
Thanks @felixcheung for review, will merge it into master if no more discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2237
`HiveConf` only means the hive jar is loaded, but doesn't mean spark is built with hive profile enabled.
`org.apache.spark.sql.hive.HiveSessionState` is private in spark and `org.apache.spark.sql.hive.execution.InsertIntoHiveTable` is public which mean it should be stable and won't be removed in spark 2.x I believe. `SparkSession.hiveClassesArePresent` is private, so I don't think it is stable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2237: ZEPPELIN-2377. Hive Support can not be enabled ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2237
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2237
I don't prefer to call private method `hiveClassesArePresent` which is not guaranteed to be stable. The cause of hive support broken in spark master is due to we copied the implementation of `hiveClassesArePresent` of spark 2.0 in zeppelin, but `hiveClassesArePresent` is changed in spark master. So instead of checking difference classes for different versions, here I propose to checking class `InsertIntoHiveTable` which is public.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2237
I guess it's fine - but we are looking fairly deep into internal stuff. Why don't we just look for HiveConf?
Or SparkSession.hiveClassesArePresent if Spark 2.0+
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2237
@Leemoonsoo @felixcheung Please help review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2237: ZEPPELIN-2377. Hive Support can not be enabled in spar...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2237
I don't disagree - we have other dependencies on private methods though?
Ok I'm fine with this approach. Thanks for the explanation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---