You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pgandhi999 <gi...@git.apache.org> on 2017/08/24 22:11:40 UTC
[GitHub] spark pull request #19047: [SPARK-21798]: No config to replace deprecated SP...
GitHub user pgandhi999 opened a pull request:
https://github.com/apache/spark/pull/19047
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server
History Server Launch uses SparkClassCommandBuilder for launching the server. It is observed that SPARK_CLASSPATH has been removed and deprecated. For spark-submit this takes a different route and spark.driver.extraClasspath takes care of specifying additional jars in the classpath that were previously specified in the SPARK_CLASSPATH. Right now the only way specify the additional jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH (https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume is a distribution classpath. It would be nice to have a similar config like spark.driver.extraClasspath for launching daemons similar to history server.
Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
## How was this patch tested?
Initially, history server start script would fail for the reason being that it could not find the required jars for launching the server in the java classpath. Same was true for running Master and Worker in standalone mode. By adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath, both the daemons(History Server, Standalone daemons) are starting up and running.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pgandhi999/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19047.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19047
----
commit 172fc20898896058b7288360eb5292ed9df9d79c
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-07-21T21:00:22Z
[SPARK-21503]: Fixed the issue
Added the case ExecutorLostFailure which was previously not there, thus, the default case would be executed in which case, task would be marked as completed.
commit 81422e0f634c0f06eb2ea29fba4281176a1ab528
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-07-25T14:54:41Z
[SPARK-21503][UI]: Adding changes as per comments
commit 55c6c37d09b41ae6914edb5d067e7f2c252ac92a
Author: pgandhi999 <pa...@gmail.com>
Date: 2017-07-26T21:26:27Z
Merge pull request #1 from apache/master
Apache Spark Pull Request - July 26, 2017
commit f454c8933e07967548095e068063bd313ae4845c
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-07-26T21:41:16Z
[SPARK-21541]: Spark Logs show incorrect job status for a job that does not create SparkContext
Added a flag to check whether user has initialized Spark Context. If it is true, then we let Application Master unregister with Resource Manager else we do not.
commit 6b7d5c6e2565c7c4dd97f31fe404c59e73c7474c
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-07-26T21:58:27Z
Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job that does not create SparkContext"
This reverts commit f454c8933e07967548095e068063bd313ae4845c.
"Merged another issue to this one by mistake"
commit bc4166490d2ff68898c00fae4c1ca1b8abe1e795
Author: pgandhi999 <pa...@gmail.com>
Date: 2017-07-28T15:24:55Z
Merge pull request #2 from apache/master
Spark - July 28, 2017
commit e46126fe0f3d8d6f92f7f51c30d8c2154bddc126
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-07-28T16:08:08Z
[SPARK-21503]- Making Changes as per comments
[SPARK-21503]- Making Changes as per comments: Removed match case statement and replaced it with an if clause.
commit 9b3cebc6b65d2da835f02efaa27015cfd1b0ccae
Author: pgandhi999 <pa...@gmail.com>
Date: 2017-08-01T13:58:12Z
Merge pull request #4 from apache/master
Spark - August 1, 2017
commit 7f03341093c843086920e8218463b5d2ba6e37d2
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-08-01T15:52:13Z
[SPARK-21503]: Reverting Unit Test Code
[SPARK-21503]: Reverting Unit Test Code - Not needed.
commit 2d01cab45ae269db9044815970dd008c851a46cc
Author: pgandhi999 <pa...@gmail.com>
Date: 2017-08-24T21:59:52Z
Merge pull request #5 from apache/master
SPARK - August 24, 2017
commit eaf63e6bd4dddc726cf57fda080b9b5d6341e2f8
Author: pgandhi <pg...@yahoo-inc.com>
Date: 2017-08-24T22:03:29Z
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server
Adding new env variable SPARK_DAEMON_CLASSPATH to set classpath for launching daemons. Tested and verified for History Server and Standalone Mode.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19047
**[Test build #81141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81141/testReport)** for PR 19047 at commit [`e421a03`](https://github.com/apache/spark/commit/e421a03acbd410a835cf3117fe6592523dc649b5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19047
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19047
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81141/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19047
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19047
LGTM. I'd like to see these daemons start using normal Spark configs like the applications do, but that's a separate, larger change...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19047
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19047
**[Test build #81141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81141/testReport)** for PR 19047 at commit [`e421a03`](https://github.com/apache/spark/commit/e421a03acbd410a835cf3117fe6592523dc649b5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19047: [SPARK-21798]: No config to replace deprecated SP...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19047
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19047: [SPARK-21798]: No config to replace deprecated SP...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19047#discussion_r135247004
--- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -136,7 +136,8 @@ void addOptionString(List<String> cmd, String options) {
Set<String> cp = new LinkedHashSet<>();
addToClassPath(cp, appClassPath);
-
+ addToClassPath(cp, getenv("SPARK_DAEMON_CLASSPATH"));
--- End diff --
I think it would be better to only support this env variable for daemon process like HistoryServer, ExternalShuffleService and others, like what we did for `SPARK_DAEMON_JAVA_OPTS`. Currently with your fix normal Spark application will also honor this env variable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19047
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19047: [SPARK-21798]: No config to replace deprecated SP...
Posted by pgandhi999 <gi...@git.apache.org>.
Github user pgandhi999 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19047#discussion_r135297693
--- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -136,7 +136,8 @@ void addOptionString(List<String> cmd, String options) {
Set<String> cp = new LinkedHashSet<>();
addToClassPath(cp, appClassPath);
-
+ addToClassPath(cp, getenv("SPARK_DAEMON_CLASSPATH"));
--- End diff --
I agree with you on this. I have made the necessary changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/19047
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19047
**[Test build #81133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81133/testReport)** for PR 19047 at commit [`e421a03`](https://github.com/apache/spark/commit/e421a03acbd410a835cf3117fe6592523dc649b5).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19047
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19047
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81133/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19047: [SPARK-21798]: No config to replace deprecated SPARK_CLA...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19047
**[Test build #81133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81133/testReport)** for PR 19047 at commit [`e421a03`](https://github.com/apache/spark/commit/e421a03acbd410a835cf3117fe6592523dc649b5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org