You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by echarles <gi...@git.apache.org> on 2015/11/01 11:47:47 UTC
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
GitHub user echarles opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/386
ZEPPELIN-383 CLASSPATH_OVERRIDES environment variable
Prepend CLASSPATH_OVERRIDES environment variable when building CLASSPATH in the shell scripts.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/echarles/incubator-zeppelin ZEPPELIN-383-CLASSPATH_OVERRIDES
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/386.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #386
----
commit 92606b736dad184d60f0225ed5de45050e8c6a63
Author: Eric Charles <er...@datalayer.io>
Date: 2015-11-01T10:44:09Z
Prepend CLASSPATH_OVERRIDES environment variable when building CLASSPATH in the shell scripts
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153572008
@Leemoonsoo: I think not exactly. `ZEPPELIN_CLASSPATH` is built by prepending elements to the classpath, meaning, whatever the user puts in the class path is, at the time of complete class path, placed at the very end of the class path. Consider a scenario:
Somebody wants to put a patched `spark-core` jar into the class path. He will set `ZEPPELIN_CLASSPATH` to his `/opt/whatever/spark-core_2.10-1.5.1.jar` and proceed with the standard procedure for `zeppelin.sh`. His jar will not be the effective one. The reason is that the class path built by Zeppelin is the following:
ZEPPELIN_CLASSPATH=<addJar>:$ZEPPELIN_CLASSPATH
...
ZEPPELIN_CLASSPATH=<addJar>:$ZEPPELIN_CLASSPATH
I believe what @echarles introduces here, is the mechanism for the user to be able to override any arbitrary functionality _no matter what_.
Very nice feature, @echarles.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-154452108
#398 replaces this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by radekg <gi...@git.apache.org>.
Github user radekg commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153724695
@Leemoonsoo that is exactly the point. JVM will use first classes found on the class path. The user can do the following:
export CLASSPATH_OVERRIDES=/opt/whatever/spark-core_2.10-1.5.1-patched-and-fixed.jar
And then it does not matter what `org.apache.spark` classes (as long as class version matches) are further down the class path: http://stackoverflow.com/a/6644467/56250
The current mechanism does not allow hot patching anything. The only patching mechanism is recompiling Zeppelin with modified JARs, which maybe difficult / not doable.
Zeppelin prepends new entries but there is no way to tell it "after all default prepends, add this at the start". And replying to your 2 comments - yes, indeed :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153723845
Oh, i misunderstood how ZEPPELIN_CLASSPATH is being populated @radekg you're right.
Then `ZEPPELIN_CLASSPATH` will append and `CLASSPATH_OVERRIDES` will prepend classpath to `CLASSPATH`. Do you think we need both?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-152999997
Thanks for the contribution.
But I think `ZEPPELIN_CLASSPATH` already does the job. isn't it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153724315
@Leemoonsoo I changed the bin/zeppelin.sh script to print the build CLASSPATH just after the execution of the java command with
```
echo CLASSPATH=$CLASSPATH
$(exec $ZEPPELIN_RUNNER $JAVA_OPTS -cp $CLASSPATH_OVERRIDES:$CLASSPATH $ZEPPELIN_SERVER "$@")
```
When I run exporting, I get
```
$ ZEPPELIN_CLASSPATH=MYJAR.JAR ./bin/zeppelin.sh
CLASSPATH=/opt/apache-ivy-2.3.0-rc1/ivy-2.3.0-rc1.jar:/opt/clojure-1.6.0/clojure-1.6.0.jar::/opt/zeppelin-incubator/zeppelin-server/target/lib/*:/opt/zeppelin-incubator/zeppelin-zengine/target/lib/*:/opt/zeppelin-incubator/zeppelin-interpreter/target/lib/*:/opt/zeppelin-incubator/*:MYJAR.JAR:/opt/zeppelin-incubator/conf:/opt/zeppelin-incubator/zeppelin-interpreter/target/classes:/opt/zeppelin-incubator/zeppelin-zengine/target/classes:/opt/zeppelin-incubator/zeppelin-server/target/classes
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-server/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-zengine/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
...
which shows the MYJAR.JAR is not taken into account, hence the need for CLASSPATH_OVERRIDES.
This is why we need a way to override by prepending, not appending.
This is useful when we deploy with e.g. a patched jar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153731822
I rebased here, but should have done for #385
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153721316
@radekg Thanks for the explanation. But i think @echarles proposed codes prepend `CLASSPATH_OVERRIDES` before `CLASSPATH`, like
```
-cp ${CLASSPATH_OVERRIDES}:${CLASSPATH}
```
That's why i think it is the same to how `ZEPPELIN_CLASSPATH` is being populated.
please correct me if my understanding is wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/386#issuecomment-153960766
I think you have merged other branches. Could you manage this PR includes only your commits?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-383 CLASSPATH_OVERRIDES ...
Posted by echarles <gi...@git.apache.org>.
Github user echarles closed the pull request at:
https://github.com/apache/incubator-zeppelin/pull/386
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---