You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2016/12/13 09:01:45 UTC
[GitHub] zeppelin pull request #1751: LIVY-1786. Refactor LivyHelper
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/1751
LIVY-1786. Refactor LivyHelper
### What is this PR for?
This PR continue the work of livy refactoring. Here's the main changes in this PR
* Move the code from `LivyHelper` to `BaseLivyInterprereter`
* Define POJO for livy request and response
* Move the livy session initialization from `interpret` to `open`
* Add one more complicated test which use the spark basic tutorial note.
* Support livy.sql for spark2
* Use zeppelin.livy.create.session.timeout instead of retry count as retry count is internal implementation and user don't know what does it mean.
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1786
### How should this be tested?
One more integration test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1786
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1751
----
commit 0043a08293799b081bcde4759f05aba2cf5f5dbc
Author: Jeff Zhang <zj...@apache.org>
Date: 2016-12-12T04:02:03Z
LIVY-1786. Refactor LivyHelper
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1751
LGTM and merge to master if there're no further discussions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1751
LGTM
CI was passing, merging if no more comments
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1751
only a couple of minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1751
@prabhjyotsingh @bzz @felixcheung Please help review.
The failed test is not relevant.
```
16/12/14 08:08:03 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
Results :
Tests in error:
DepInterpreterTest.testDefault:96 � NullPointer
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1751
Please hold on the merge. I find another bug in livy interpreter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1751
Merging this if no more discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1751
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1751
@felixcheung The new bug is fixed. CI fail is not relevant.
```
04:42:36,519 ERROR org.apache.zeppelin.search.LuceneSearch:120 - Failed to open index dir RAMDirectory@615184ee lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2eee572c, make sure indexing finished OK
org.apache.lucene.index.IndexNotFoundException: no segments* file found in RAMDirectory@615184ee lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2eee572c: files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:726)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:50)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
at org.apache.zeppelin.search.LuceneSearch.query(LuceneSearch.java:104)
at org.apache.zeppelin.search.LuceneSearchTest.canNotSearchBeforeIndexing(LuceneSearchTest.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
```
```
Results :
Failed tests:
ZeppelinSparkClusterTest.zRunTest:347 expected:<FINISHED> but was:<ERROR>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu closed the pull request at:
https://github.com/apache/zeppelin/pull/1751
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1751
Just a minor recommendation, rest LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1751
ping @prabhjyotsingh @bzz @felixcheung @Leemoonsoo
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1751
@Leemoonsoo @felixcheung Thanks for review, comments are addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1751: ZEPPELIN-1786. Refactor LivyHelper
Posted by zjffdu <gi...@git.apache.org>.
GitHub user zjffdu reopened a pull request:
https://github.com/apache/zeppelin/pull/1751
ZEPPELIN-1786. Refactor LivyHelper
### What is this PR for?
This PR continue the work of livy refactoring. Here's the main changes in this PR
* Move the code from `LivyHelper` to `BaseLivyInterprereter`
* Define POJO for livy request and response
* Move the livy session initialization from `interpret` to `open`
* Add one more complicated test which use the spark basic tutorial note.
* Support livy.sql for spark2
* Use zeppelin.livy.create.session.timeout instead of retry count as retry count is internal implementation and user don't know what does it mean.
* Improve travis (wrap livy related work in `setupLivy.sh`)
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-1786
### How should this be tested?
One more integration test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-1786
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1751
----
commit 0a1aa5dd993141efb6034b4799ba8aebec1a80dc
Author: Jeff Zhang <zj...@apache.org>
Date: 2016-12-12T04:02:03Z
ZEPPELIN-1786. Refactor LivyHelper
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---