You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by prabhjyotsingh <gi...@git.apache.org> on 2016/08/11 15:16:43 UTC
[GitHub] zeppelin pull request #1322: [ZEPPELIN-1320] Security fix for Shell/Spark an...
GitHub user prabhjyotsingh opened a pull request:
https://github.com/apache/zeppelin/pull/1322
[ZEPPELIN-1320] Security fix for Shell/Spark and Python Interpreter
### What is this PR for?
While running a Notebook using shell, spark, python uses same user as which zeppelin server is running. Which means these interprets have same permission on file system as zeppelin server.
IMO user should have option to run these interpreters as different user.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Update doc
### What is the Jira issue?
* [ZEPPELIN-1320](https://issues.apache.org/jira/browse/ZEPPELIN-1320)
### How should this be tested?
- Add an user in system say "zeppelin-interpreter"
- Add ssh key for the same
```
ssh-keygen
ssh zeppelin-interpreter@localhost mkdir -p .ssh
cat ~/.ssh/id_rsa.pub | ssh zeppelin-interpreter@localhost 'cat >> .ssh/authorized_keys'
```
- Add `export ZEPPELIN_INTERPRETER_USER="zeppelin-interpreter"` in `zeppelin-env.sh`
- Start zeppelin server, try and run following in paragraph in a notebook
```
%sh
whoami
```
Check that it should run as new user, i.e. "zeppelin-interpreter"
### Screenshots (if appropriate)
<img width="1440" alt="screen shot 2016-08-11 at 8 45 12 pm" src="https://cloud.githubusercontent.com/assets/674497/17593747/8c9eb096-6004-11e6-8487-3e44a1a0d6eb.png">
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prabhjyotsingh/zeppelin ZEPPELIN-1320
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1322.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1322
----
commit 6aafac445db4cd8944caa07280fcdbebb2faea0f
Author: Prabhjyot Singh <pr...@gmail.com>
Date: 2016-08-11T15:08:12Z
user should have option to run these interpreters as different user.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Closing this, will open a new one with merge of https://github.com/apache/zeppelin/pull/1265.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Instead of `USE_SSH_IMPERSONATION`, how about let user customize impersonation method?
For example,
```
ZEPPELIN_INTERPRETER_IMPERSONATION_CMD="su - ${ZEPPELIN_USER_NAME}"
```
by default. but user can override this env variable, like
```
ZEPPELIN_INTERPRETER_IMPERSONATION_CMD="ssh -p12345 ${ZEPPELIN_USER_NAME}@localhost"
```
It gives more flexibility i think. (e.g. give additional options like -p. use different command to impersonate)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by astroshim <gi...@git.apache.org>.
Github user astroshim commented on the issue:
https://github.com/apache/zeppelin/pull/1322
I got following checkstyle error while building source.
```
[INFO] There are 1 checkstyle errors.
[ERROR] NotebookServer.java[1381] (sizes) LineLength: Line is longer than 100 characters (found 102).
```
@prabhjyotsingh Could you fix this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@zjffdu Yes, you are right, with SPARK_HOME/SPARK_SUBMIT it doesn't work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Agreed @echarles, the `doAs` part will be a problem, until [ZEPPELIN-1340](https://issues.apache.org/jira/browse/ZEPPELIN-1340) is resolved. Until then for security we may have to run half interpreter with "User Impersonate" enable from UI (for example shell, python interpreter), and for others use the standard `doAs` already implemented (like livy, spark, jdbc)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Yes, you are right, let me do it right away.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@echarles , Yes agreed, will need to update in doc, and a extra toolbar near the check box where user can enable User Impersonate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Security fix for Shell/Spark and Pytho...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1322
shouldn't interpreter process be impersonating the user logging onto the web front end?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@prabhjyotsingh Without issues above, Could you check this PR support `scoped` as well which uses multiple threads in one process?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
CI green! Ready for review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Sure, In this PR I was only thinking about the otherwise case i.e. in the environment where hadoop dependencies where not present, and hence start interpreter as end-web-user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Whatever `su` or `ssh` is used, I feel the main trick is the user provisioning on the host running the interpreter. Until now, the shiro authentication system had no impact on the user provisioning. This PR changes this.
I guess we all agree and are aware that adding user `foo` to shiro.ini, and enabling impersonation, will require to run `adduser foo` manually.
We should make this clear in the doc but also stress it in the UI (with a hover, or a clear text/link near the <checkbox> User Impersonate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Yes, I thought about the usage in mac and windows, and initially started of with using `RUNAS ${userName}` for windows and `su - ${userName}` for *nix systems, but then it requires zeppelin server to run as root. Hence, implemented with `ssh ${userName}@localhost`.
Have not thought about the cases in which system admin uses virtual users.
Now since with this, we are able to propagate end web user to RemoteInterpreterManagedProcess.start, we can choose to use some other mechanism in `interpreter.sh`/`interpreter.cmd` instead of "ssh", or may be make it configurable using some extra config in "zeppelin-env.sh"
What do you recommend, that would be a secure and all full proof mechanism by which we can run interpreter as different user ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@Leemoonsoo yes thats a good suggestion. Let me try and do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1322: [ZEPPELIN-1320] Run zeppelin interpreter proces...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh closed the pull request at:
https://github.com/apache/zeppelin/pull/1322
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/1322
... and make ZEPPELIN-1320 a subtask of ZEPPELIN-1337
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1322
I agree that it's simple way to use ssh to support impersonation. but I'm worried about it. First, we should consider not to use ssh server in a local machine. It's disabled on Mac by default and in case of Windows users, they might not have any ssh server. Second, even if all of users can use connect their machine via ssh, all of users' name should be the same as system users. AFAIK, Some Zeppelin use cases, the system admin uses virtual users as well. Do you think of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@prabhjyotsingh I agree @echarles's idea. Interpreter tries to find hadoop dependencies first and if it passes, it uses `doAs`. Otherwise, let's talk about how to do it. How do you think of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Then I think we should either revert this PR or fix it for spark interpreter as well. Because spark interpreter is the most important interpreter of zeppelin IMO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Sorry for late comment. I was in vacation in the last 2 weeks. I found this didn't work for spark interpreter. @prabhjyotsingh Did you try it for spark interpreter and other interpreters ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
> It's better to use RUNAS ~ and su - ~
@jongyoul How about I make `use RUNAS ~ and su - ~` by default, but if in `zeppelin-env.sh` a property say `USE_SSH_IMPERSONATION` is set to true, then it will use `ssh web-user@localhost` in this way user gets to decide, what is best suited for their user case.
>Could you check this PR support `scoped` as well which uses multiple threads in one process?
Yes I've checked this with Shell and Python interpreter it was working as expected.
@Leemoonsoo, yes agreed, I too think this options should be there, and have implemented it as well. If you take a look at GIF attached in this PR description, it's doing that you are asking for :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/1322
To make ZEPPELIN-1337 Umbrella for multiple user support for zeppelin more readable, should we rename the following:
- ZEPPELIN-1340: "Run Hadoop-based interpreter process on Kerberos as web front end user"
- ZEPPELIN-1320: "Run zeppelin interpreter process as web front end user"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by jongyoul <gi...@git.apache.org>.
Github user jongyoul commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@prabhjyotsingh I don't know how to support different users' environments fully, actually. But I think it's better to use `RUNAS ~` and `su - ~` and using `ssh` without password make some security issues. In case Mesos, it uses that way to support restrict resources. But I never see using `ssh` without password. How do you think of it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1322
If i add one more,
What do you guys think about adding an option `Impersonate` in the interpreter setting on GUI?
That'll give user flexibility of selecting current behavior (without impersonation) and new behavior. Otherwise, this PR will make incompatible user behavior change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Security fix for Shell/Spark and Pytho...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1322
@felixcheung Fair point, let me try and do it, will change the title to WIP for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1322: [ZEPPELIN-1320] Run zeppelin interpreter process as we...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the issue:
https://github.com/apache/zeppelin/pull/1322
Btw, for the hadoop case (or spark on yarn case), this PR may give an issue for `doAs`.
Typically, you configure `hadoop.proxyuser.foo.hosts` and `hadoop.proxyuser.foo.group`, `foo` being the os/kerberos user under which you run your java code that calls doAs.
If we run ssh/su as the front-end user, we will not fullfill what the hadoop/yarn cluster is expecting.
We thus should have two checkboxes:
- One for the OS/kerberos impersonation (this PR only adresses OS).
- The other for Hadoop impersonation.
If you select one, I would expect the other one to be disabled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---