You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2017/04/07 07:13:37 UTC
[GitHub] zeppelin pull request #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.sp...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2231
ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress() for the spark master
### What is this PR for?
`SparkUI.addUIAddress` is removed in spark master which cause this error. Actually spark 2.0 introduce new api `SparkContext.uiWebUrl` (SPARK-14576) which is public in contrast `SparkUI.addUIAddress` is private. This PR would use `SparkUI.addUIAddress` for spark 1 and `SparkContext.uiWebUrl` for spark 2.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2150
### How should this be tested?
Manually verify it on spark 1.6.2, spark 2.1.0 and spark master
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2150
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2231.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2231
----
commit beb7925bbaff31e1a3361e8204cfc16bd3ecd26b
Author: Jeff Zhang <zj...@apache.org>
Date: 2017-02-22T11:11:24Z
ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress() for the spark master
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by thesuperzapper <gi...@git.apache.org>.
Github user thesuperzapper commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu what version of spark were you using, as this only occurs >=2.2
It looks like we still try to call (the now removed as of Spark 2.2) "appUIAddress":
https://github.com/apache/zeppelin/blob/master/livy/src/main/java/org/apache/zeppelin/livy/LivySparkInterpreter.java#L51
Meaning its pretty likely that this issue still occurs in master for >=2.2. (I cant check in my environment)
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by qwemicheal <gi...@git.apache.org>.
Github user qwemicheal commented on the issue:
https://github.com/apache/zeppelin/pull/2231
This problem still exist in LivyPySparkBaseInterpreter.java. This file still use sc._jsc.sc().ui().get().appUIAddress() and %livy.pyspark doesn't work when using spark 2.2.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by yywwd <gi...@git.apache.org>.
Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
I have solved this problem by editing some configurations in Livy. SivaKaviyappa's solution: setting "livy.spark.master yarn-cluster" does not work for me. I do not edit "livy.spark.master" in livy.conf. Instead, I uncomment "livy.spark.deploy-mode" and set the value to cluster. The edited configuration is:
# What spark deploy mode Livy sessions should use.
livy.spark.deploy-mode cluster
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/2231
LGTM, manually verify it on spark 1.6.1, spark 2.0.0 and spark master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by liusztc09 <gi...@git.apache.org>.
Github user liusztc09 commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu
I run into the same issue with Livy 0.4 and Spark 2.2.1. There is a comment by you saying it's only affecting livy 0.2 in https://issues.apache.org/jira/browse/ZEPPELIN-3269. Not sure whether it's fixed yet.
Seems the old method was still being called.
https://github.com/apache/zeppelin/search?utf8=%E2%9C%93&q=appUIAddress%28%29+&type=
Any workarounds or ETA on the fix?
Thanks!
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@felixcheung @karuppayya please help review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by yywwd <gi...@git.apache.org>.
Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress()
My AWS cluster is EMR-5.14.0, Application: Ganglia 3.7.2, Spark 2.3.0, Zeppelin 0.7.3, Livy 0.4.0
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by DerekKaknes <gi...@git.apache.org>.
Github user DerekKaknes commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Thanks for following-up. I'm running Livy connected to a standalone cluster, but in digging deeper it's possible that there is an issue with Livy always configuring itself to local mode when building spark sessions. I'll have to look at that first to see if it resolves this issue as well.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Could you share your code that use programmatic API ?
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by ererermao <gi...@git.apache.org>.
Github user ererermao commented on the issue:
https://github.com/apache/zeppelin/pull/2231
I also encountered this problem on aws (emr 5.13.0).
zeppelin(0.7.3)、livy(0.4.0)、saprk(2.3.0)
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by yywwd <gi...@git.apache.org>.
Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu I change the yarn mode into "yarn-cluster" as SivaKaviyappa suggested, Zeppelin works well. However, when I use programmatic API, it will throw such exception: ImportError: No module named cloudpickle.cloudpickle. If I change the yarn mode into default mode, that is "yarn", programmatic API will works well, but Zeppelin still cannot work. My cluster is EMR-5.14.0, Application: Ganglia 3.7.2, Spark 2.3.0, Zeppelin 0.7.3, Livy 0.4.0
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
That method would only be called in livy 0.2. In livy 0.2 the spark web url will be returned from livy rest api, so that piece of code won't be invoked. And zeppelin has system test for that, not sure why you would hit this kind of issue.
Are you running livy in local mode ? It is suggested to run livy in yarn-cluster mode.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Thanks for review. @prabhjyotsingh
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/2231
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Thanks for reporting, created ZEPPELIN-3269
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Thanks @SivaKaviyappa for confirming
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by DerekKaknes <gi...@git.apache.org>.
Github user DerekKaknes commented on the issue:
https://github.com/apache/zeppelin/pull/2231
I'm having this same issue with zeppelin 0.7.3 + livy 0.5.0 + spark 2.2.1. I have not yet verified that zeppelin master resolves this issue, but from inspection I would think that @thesuperzapper is right that the issue would persist. I would think that any invocation of `SparkUI.appUIAddress()` would throw a `NoSuchMethodException`, and zeppelin master still seems to be making those calls (specifically from `LivySparkInterpreter`). @zjffdu Do you have any notion as to why your interpreter would not be throwing an exception?
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/2231
LGTM
Is "" handled properly when returned?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Good catch! @felixcheung It would be better to return null, fixed that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by rouckas <gi...@git.apache.org>.
Github user rouckas commented on the issue:
https://github.com/apache/zeppelin/pull/2231
I was also running into this issue with Zeppelin 0.7.3, Livy 0.5.0 (0.4.0), and Spark 2.3.0 (or 2.2.1). The only solution I found was to downgrade to Zeppelin 0.6.2.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by SivaKaviyappa <gi...@git.apache.org>.
Github user SivaKaviyappa commented on the issue:
https://github.com/apache/zeppelin/pull/2231
This is works for me on aws emr. I am on emr 5.13.0, livy 0.40 , spark 2.3.0. The important change one has to make is by default aws emr brings up livy in yarn mode. So if you change that yarn-cluster mode and stop livy server, zeppelin (emr automatically restarts the services once stopped) then this above issue does not happen. Thanks to @zjffdu for pointing that out.
# What spark master Livy sessions should use.
livy.spark.master yarn-cluster
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@thesuperzapper @ererermao It works for me on zeppelin master + livy 0.5.0 + spark 2.3.0
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
Sorry, I still not sure what code you run in zeppelin. If it is livy related issue, you need to ask question in livy mail list.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by thesuperzapper <gi...@git.apache.org>.
Github user thesuperzapper commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu @felixcheung @ererermao there seems to be no traction in this, it hasn't worked for a long time, https://issues.apache.org/jira/browse/ZEPPELIN-3455.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
What kind of error you see in zeppelin when using livy.spark.master yarn mode
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by yywwd <gi...@git.apache.org>.
Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu This issue is not related to the codes run in zeppelin. I mean, I cannot guarantee “Using Livy interpreter in Zeppelin” and "Using Programmatic API" work well at the same time.
Using Zeppelin, I just write codes like this:
```
%livy.pyspark
sc.version
```
just test whether it works.
Using Programmatic API, I just use official examples: https://github.com/apache/incubator-livy/blob/master/examples/src/main/python/pi_app.py
**BUT!!!**
In **livy.spark.master yarn-cluster** mode zeppelin works, programmatic API cannot work
In **livy.spark.master yarn** mode (the default mode), programmatic API works, zeppelin cannot work
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by SivaKaviyappa <gi...@git.apache.org>.
Github user SivaKaviyappa commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@yywwd - Few things to check
What are your zeppelin-env.sh and livy intrepreter settings
1.have you set the PYTHONPATH in the zeppelin-env.sh
2. zeppelin.livy.url=http://localhost:8998 - have set this value in your livy intrepreter?
3. you have installed python libraries in all the core nodes?
I am using emr 5.11.0 and everything works fine.
---
[GitHub] zeppelin pull request #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.sp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2231
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@rouckas I try zeppelin master & livy 0.5, it works for me. Do you mind to try it in zeppelin master ?
---
[GitHub] zeppelin issue #2231: ZEPPELIN-2150. NoSuchMethodError: org.apache.spark.ui....
Posted by yywwd <gi...@git.apache.org>.
Github user yywwd commented on the issue:
https://github.com/apache/zeppelin/pull/2231
@zjffdu I'm sorry to response slowly, because I tried to reproduce the bugs. I thought it may caused by my codes, so I tried to use official PySpark examples, the bugs still occurred. This is the code I used. https://github.com/apache/incubator-livy/blob/master/examples/src/main/python/pi_app.py
**Note:** I comment the last line # client.stop(True), cause I won't want to close the session after submitting just one job. The details about this bug are as follows:
1. When I use default yarn mode, that is "yarn", official PySpark examples and programmatic API work well. Using Livy interpreter in Zeppelin will throw exception:` NoSuchMethodError: org.apache.spark.ui.SparkUI.appUIAddress()` for the spark master.
2. ThenI change the yarn mode into "yarn-cluster" as SivaKaviyappa suggested, Zeppelin works well. But the logs of this statement will hav a warning: `"Warning: Master yarn-cluster is deprecated since 2.0. Please use master \"yarn\" with specified deploy mode instead.`
However, Using programmatic API will have such bug:
2.1 I delete all Livy sessions, and run pi_app.py. It throw such exception:
```
ReadTimeout: HTTPConnectionPool(host='172.31.5.251', port=8998): Read timed out. (read timeout=10)
Traceback (most recent call last):
File "/home/ec2-user/wandongwu/livy_test_9/pi_app.py", line 35, in <module>
pi = client.submit(pi_job).result()
File "/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
return self.__get_result()
File "/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
raise exception_type, self._exception, self._traceback
TypeError: raise: arg 3 must be a traceback or None
```
But I find It has started a new Livy session. So I edit configuration parameter into `'http://<livy-server>:8998/sessions/0' 2`. It will throw another exception:
```
Traceback (most recent call last):
File "/home/ec2-user/wandongwu/livy_test_9/pi_app.py", line 35, in <module>
pi = client.submit(pi_job).result()
File "/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
return self.__get_result()
File "/usr/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
raise exception_type, self._exception, self._traceback
Exception: org.apache.livy.repl.PythonJobException: Client job error:Traceback (most recent call last):
File "/mnt/yarn/usercache/livy/appcache/application_1528945006613_0302/container_1528945006613_0302_01_000001/tmp/4991895008696585180", line 159, in processBypassJob
deserialized_job = pickle.loads(serialized_job)
File "/usr/lib64/python2.7/pickle.py", line 1388, in loads
return Unpickler(file).load()
File "/usr/lib64/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/usr/lib64/python2.7/pickle.py", line 1096, in load_global
klass = self.find_class(module, name)
File "/usr/lib64/python2.7/pickle.py", line 1130, in find_class
__import__(module)
ImportError: No module named cloudpickle.cloudpickle
```
3. Then I change yarn mode into default mode, that is "yarn", programmatic API can work well, but Zeppelin still cannot work.
---