You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by riverajo <gi...@git.apache.org> on 2015/08/28 20:41:47 UTC
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
GitHub user riverajo opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/264
ZEPPELIN-270 added env varibles for pyspark's correct function in yarn.
Simple doc fix for pyspark in yarn.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/riverajo/incubator-zeppelin gh-pages
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/264.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #264
----
commit 03c80a091cb5959a50d8e598e8088a406468a67e
Author: Joseph D Rivera <jo...@us.army.mil>
Date: 2015-08-28T18:38:26Z
ZEPPELIN-270 added env varibles for pyspark's correct function in yarn.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-136437760
@felixcheung
> Is this a bug:
> ```
if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
conf.set(key, val);
}
```
> Shouldn't it say `if (key.startsWith("spark.")`?
I think it's not a bug. Designed to not pass empty value when the key starts with "spark."
This unittest might helpful to understand.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java#L168
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-135927106
I think PYTHONPATH supposed to build automatically when SPARK_HOME is defined by [bin/interpreter.sh](https://github.com/apache/incubator-zeppelin/blob/master/bin/interpreter.sh#L77). But SPARK_YARN_USER_ENV is not taken care of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by Leemoonsoo <gi...@git.apache.org>.
Github user Leemoonsoo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-136438850
> Hmm, should we use SparkConf instead then:
>
> https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
> ```
// Keep this for backwards compatibility but users should move to the config
sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
}
```
> I believe all spark.* in the interpreter settings are passed to SparkConf.
How about put the same comment `Keep this for backwards compatibility but users should move to the config` in the document and proceed?
How about take care [zeppelin-env.sh](https://github.com/apache/incubator-zeppelin/blob/master/conf/zeppelin-env.sh.template#L40), too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-135905387
It should build this automatically when SPARK_HOME is defined right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-136522014
@Leemoonsoo good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by riverajo <gi...@git.apache.org>.
Github user riverajo commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-136382821
> It should build this automatically when SPARK_HOME is defined right?
That seems right, I didn't need to set PYTHONPATH if SPARK_HOME is set correctly, but I do still need SPARK_YARN_USER_ENV.
However it seems as thought the PYTHONPATH gets created after the zeppelin-env.sh gets run, If I don't set both PYTHONPATH and SPARK_YARN_USER_ENV in that file, It complains that SPARK_YARN_USER_ENV is empty with an array out of bounds exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: ZEPPELIN-270 added env varibles f...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/264#issuecomment-135931941
Hmm, should we use SparkConf instead then:
https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
```
// Keep this for backwards compatibility but users should move to the config
sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
}
```
I believe all spark.* in the interpreter settings are passed to SparkConf.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
Is this a bug:
```
if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
conf.set(key, val);
}
```
Shouldn't it say `if (key.startsWith("spark.")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---