You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by prabhjyotsingh <gi...@git.apache.org> on 2016/11/23 07:42:57 UTC
[GitHub] zeppelin pull request #1672: ZEPPELIN-1701: User impersonation not work with...
GitHub user prabhjyotsingh opened a pull request:
https://github.com/apache/zeppelin/pull/1672
ZEPPELIN-1701: User impersonation not work with SPARK_HOME set.
### What is this PR for?
User impersonation of front end user does not work with SPARK_HOME set.
### What type of PR is it?
[Bug Fix | Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* [ZEPPELIN-1701](https://issues.apache.org/jira/browse/ZEPPELIN-1701)
### How should this be tested?
set SPARK_HOME zeppelin-env.sh and then try to run
```
%spark
System.getProperty("user.name")
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prabhjyotsingh/zeppelin ZEPPELIN-1701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1672.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1672
----
commit 62d4eb8027fdea8e486def849533c4354a2c9d98
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-23T07:35:19Z
ZEPPELIN-1701: User impersonation of front end user does not work with SPARK_HOME set.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
can you try
`chmod 777 /Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
It still doesn't work for me.
This is the error I get, it seems permission issue because it use another user to run the interpreter process.
```
DEBUG [2016-11-23 16:13:06,100] ({Exec Stream Pumper} RemoteInterpreterManagedProcess.java[processLine]:189) - java.io.FileNotFoundException: /Users/jzhang/github/zeppelin/logs/zeppelin-interpreter-spark-jzhang-jzhangMBPr.local.log (Permission denied)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1672: ZEPPELIN-1701: User impersonation not work with...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/1672
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by r-kamath <gi...@git.apache.org>.
Github user r-kamath commented on the issue:
https://github.com/apache/zeppelin/pull/1672
LGTM \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
Merging this if no more discussion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
@prabhjyotsingh It works now, but log file permission issue is still here which make the spark interpreter fail to run unless user manually set permission
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
@zjffdu did you try it again, as this is working for me, or if you can let me know your environment I can try on the same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
I think that should be fine;
```
...
ZEPPELIN_INTP_CLASSPATH+=":${HADOOP_CONF_DIR}"
...
CLASSPATH+=":${ZEPPELIN_INTP_CLASSPATH}"
...
SPARK_SUBMIT_RUN=` echo ${SPARK_SUBMIT} --class ${ZEPPELIN_SERVER} --driver-class-path \"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${CLASSPATH}\" --driver-java-options \"${JAVA_INTP_OPTS}\" ${SPARK_SUBMIT_OPTIONS} ${SPARK_APP_JAR} ${PORT}`
...
```
https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L171
https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L186
https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR189
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
Thank you for the verification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the issue:
https://github.com/apache/zeppelin/pull/1672
https://github.com/apache/zeppelin/pull/1704 changes the filename processenduser.md to userimpersonation.md. Could you resolve conflict?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
Actually it doesn't work for spark interpreter if you try that. https://github.com/apache/zeppelin/blob/master/bin/interpreter.sh#L171 is for pig. And it is not safe if we add other enviroment variables that is not in classpath. Could we run zeppelin-env.sh after ssh ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
I assumed this log file should have group permission to log into this file. What do you recommend ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1672: ZEPPELIN-1701: User impersonation not work with...
Posted by prabhjyotsingh <gi...@git.apache.org>.
GitHub user prabhjyotsingh reopened a pull request:
https://github.com/apache/zeppelin/pull/1672
ZEPPELIN-1701: User impersonation not work with SPARK_HOME set.
### What is this PR for?
User impersonation of front end user does not work with SPARK_HOME set.
### What type of PR is it?
[Bug Fix | Hot Fix]
### Todos
* [x] - Fix log file permission issue
### What is the Jira issue?
* [ZEPPELIN-1701](https://issues.apache.org/jira/browse/ZEPPELIN-1701)
### How should this be tested?
set SPARK_HOME zeppelin-env.sh and then try to run
```
%spark
System.getProperty("user.name")
```
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N/A
* Is there breaking changes for older versions? N/A
* Does this needs documentation? Yes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prabhjyotsingh/zeppelin ZEPPELIN-1701
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1672.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1672
----
commit 62d4eb8027fdea8e486def849533c4354a2c9d98
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-23T07:35:19Z
ZEPPELIN-1701: User impersonation of front end user does not work with SPARK_HOME set.
commit d3d020c8a3faef00bba41617e51f7df73ee50dec
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-24T11:01:27Z
execute env variables in zeppelin-env.sh
commit aa5de22d6eadccec69f0ae9bc0aaa57353e8587e
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-25T09:19:43Z
use different log file for different user (in case of user impersonation)
commit 9052f2d56d6075754c5ab745c96ab22d1e083197
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-28T05:55:14Z
Merge remote-tracking branch 'origin/master' into ZEPPELIN-1701
commit 002950bee03e24b1182abdb60c069bba395dfc50
Author: Prabhjyot Singh <pr...@gmail.org>
Date: 2016-11-28T10:16:52Z
adding missing import
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
I see your point, let me check that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
@zjffdu Here is what I've done
- https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR195
`source "${ZEPPELIN_CONF_DIR}'/zeppelin-env.sh;`
This should take care of zeppelin-env variables
- https://github.com/apache/zeppelin/pull/1672/files#diff-8d3bd42a5de342ebad38e1d113d9f75cR93
Append user name in log file, this has an assumption that the parent directory has right permission for this guy to write (typically have group permission to write and have the impersonating user in that group).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
@Leemoonsoo resolved conflict. Thank you for the quick fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
@prabhjyotsingh I can not run pyspark interpreter. Can you try that in your enviroment ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1672: ZEPPELIN-1701: User impersonation not work with...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh closed the pull request at:
https://github.com/apache/zeppelin/pull/1672
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
I found another crtitical issue is that all the enviroment variables defined in zeppelin-env.sh (like HADOOP_CONF_DIR) won't be passed to the interpreter process because you use ssh. This will cause interpreter process fail to start.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
I can do that, but it doesn't make sense to force user to do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/1672
Please ignore it\uff0cI can ran it successfully now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by prabhjyotsingh <gi...@git.apache.org>.
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1672
This is ready for review, have checked this in few flavours of linux, and it was working as expected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1672: ZEPPELIN-1701: User impersonation not work with SPARK_...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/zeppelin/pull/1672
as an aside, I'd update the JIRA/PR - it might not be obvious what "when SPARK_HOME is set" means. If external Spark is broken then I think it is a much much serious problem :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---