You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by jongyoul <gi...@git.apache.org> on 2017/06/17 13:04:37 UTC
[GitHub] zeppelin pull request #2418: [ZEPPELIN-2645] Adding way to register RemoteIn...
GitHub user jongyoul opened a pull request:
https://github.com/apache/zeppelin/pull/2418
[ZEPPELIN-2645] Adding way to register RemoteInterpreterServer's port into InterpreterProcess
### What is this PR for?
This is a thrift version related to #2410
### What type of PR is it?
[Feature]
### Todos
* [x] - Add opposite set of server/client
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2645
### How should this be tested?
N/A
### Screenshots (if appropriate)
N/A
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jongyoul/zeppelin ZEPPELIN-2645-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2418
----
commit b5f1816a65ff217c4eef6439c0084f12e3aab601
Author: Jongyoul Lee <jo...@gmail.com>
Date: 2017-06-14T10:43:49Z
Added getters
commit e0af12abcc79ecf164a40b647ea5f4fcff91f35a
Author: Jongyoul Lee <jo...@gmail.com>
Date: 2017-06-14T13:23:10Z
changed constructor for ZeppelinDevServer
commit 82236fba457db849c63c1d02ec2a1aa1d6718526
Author: Jongyoul Lee <jo...@gmail.com>
Date: 2017-06-17T00:44:23Z
Shaded netty-all dependencies into zeppelin-interpreter
commit dddd9234d28139c4353fe5b74af17d0f1aaca84b
Author: Jongyoul Lee <jo...@gmail.com>
Date: 2017-06-17T10:47:01Z
Changed netty to thrift
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/2418
I'd like adding more points to the discussion:
- Do we need to support kerberos-like auth in the communication protocol (you can achieve this with Thrift, not sure with Netty)?
- Do we target a single implementation dropping the other or something that can be configured with the burden to maintain multiple implementations?
- What about REST (for more easy and universal clients) or WebSocket (for push) as more alternatives?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@zjffdu Agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
>> Do we need to support kerberos-like auth in the communication protocol (you can achieve this with Thrift, not sure with Netty)?
This is internal communication channel between zeppelin server and interpreter process, so I don't think it is necessary to use kerberos here. But definitely, we need to make sure this channel secure, I think both thrift and netty support security channel.
>> Do we target a single implementation dropping the other or something that can be configured with the burden to maintain multiple implementations?
We would use just one, that's why I mention in the above comment that before we choose thrift or netty, we need to consider the whole picture.
>> What about REST (for more easy and universal clients) or WebSocket (for push) as more alternatives?
I don't see much advantages of using websocket. Netty and thrift both could do the push. And the communication channel here is one-to-one, not one-to-many.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/2418
In case of SNPEGO Web Auth (not yet supported by Zeppelin0,, you could consider transfert the Kerberos ticket from front-end to back-end and use it to , this the usecase I was thinking to.
Well, multiple and pluggable communication between frontend and interpreters would be great and useful, but I am aware of the impact on code level.
If we have to move to something else than thrift, what would be the added value of netty (just asking).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #2418: [ZEPPELIN-2645] Adding way to register RemoteIn...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul closed the pull request at:
https://github.com/apache/zeppelin/pull/2418
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@echarles Thanks for the feedback. Currently Interpreter Process can only communicate with Zeppelin Server process, and it is a internal communication channel which is not exposed to users and other components. Regarding your comments `Those clients could be the current javascript of Zeppelin webapp, or any other kind of client (could be bash or python script...)`, Could you give some scenarios that Interpreter Process needs to communicate with other clients ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@zjffdu Sounds good. I totally agree with your point of view. We need to add pushing way from interpreter to server, and also abstract that layer carefully to guess most of situation. This is a kind of PoC level. I'll make a design docs for handling it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@jongyoul Thanks for the PR.
1. It looks there will be one server per RemoterInterpreterProcess, will it been shutdown after it get the port and host info ? And what will happens if the interpreter process fails to launch ?
2. What is the advantage of netty over thrift ? Since this relates with the how we communicate between zeppelin server and interpreter process, we need to consider which one is better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
For that purpose, you can use zeppelin's rest api or note's cron scheduler
https://zeppelin.apache.org/docs/0.8.0-SNAPSHOT/usage/rest_api/notebook.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@echarles Hi, at first, if we enable front-end to access to interpreter directly, we have to consider that each interpreter has to know whole ACL information and etc. I don't think it's good design. Second, netty has a benefit while making connections between server and interpreter. It can be handled one connection for bidirectional communication.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@zjffdu I am thinking to automation (using the notes in an scheduled or batch process). Does it make sense to you?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@jongyoul sorry, I was not clear enough. With frontend, I was meaning the web layer (so jetty). Jetty could communicate via a variety of protocols to the interpreters. Now going further in that direction, having protocols like REST or WebSocket would allow any king of clients to access the interpeters in direct - Those clients could be the current javascript of Zeppelin webapp, or any other kind of client (could be bash or python script...)
This is unique opportunity to extend the Zeppelin user base - Btw, having the AAA (Authentication, Authorization, Auditing) functions at Interpreter level is IMHO a good design, even at the cost of some extra development or refactoring.
I am curious to know what the community thinks about this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
@jongyoul I think there's 2 follow up things to do.
* Unify the communication frame between zeppelin server & interpreter process.
* Change the pull mode to push mode so that interpreter process can push data to zeppelin-server.
When we consider to choose netty or thrift, we need to consider these 2 things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #2418: [ZEPPELIN-2645] Adding way to register RemoteInterpret...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2418
Sorry @jongyoul I did a investigation on netty, it looks like we could use Channel directly. In that way I would prefer to use netty. But still I think we need to have a good design for the communication framework change from thrift to netty considering the 2 things I mentioned above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---