You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pgandhi999 <gi...@git.apache.org> on 2017/10/02 16:11:44 UTC
[GitHub] spark pull request #19411: [SPARK-22185] : Jobs running on Spark Standalone ...
GitHub user pgandhi999 opened a pull request:
https://github.com/apache/spark/pull/19411
[SPARK-22185] : Jobs running on Spark Standalone Mode cannot access HDFS
Refer https://issues.apache.org/jira/browse/SPARK-21798.
Basically, the environment variable SPARK_DAEMON_CLASSPATH was added to replace SPARK_CLASSPATH for launching daemons. However, Spark Standalone mode is unable to run spark jobs accessing hdfs cluster as the jars required by hadoop like hadoop-gpl-compression jar exported with SPARK_DAEMON_CLASSPATH are not available in the executor launched by worker process.
## How was this patch tested?
Ran HDFS Job on Spark Standalone Mode and verified that the job properly ran.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pgandhi999/spark SPARK-22185
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19411.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19411
----
commit 5ef067238cd98f40ea140c00a3cc3013c380adce
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-10-02T16:06:43Z
[SPARK-22185] : Adding SPARK_DAEMON_CLASSPATH to class path
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19411: [SPARK-22185] : Classpath cannot be set for jobs running...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on the issue:
https://github.com/apache/spark/pull/19411
@vanzin @srowen Got the point. Will close this PR then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19411: [SPARK-22185] : Classpath cannot be set for jobs running...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19411
No. The daemon's classpath should not be reflected in the executors launched by the daemon. It can contain unwanted libraries, unwanted configuration, and other things that executors shouldn't know about.
Fix your configuration so your needed jars are in the executor's classpath too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19411: [SPARK-22185] : Jobs running on Spark Standalone Mode ca...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19411
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19411: [SPARK-22185] : Classpath cannot be set for jobs running...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/19411
This is building a user app process classpath right? it shouldn't include the daemon classpath
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19411: [SPARK-22185] : Classpath cannot be set for jobs ...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 closed the pull request at:
https://github.com/apache/spark/pull/19411
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org