You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by gzm55 <gi...@git.apache.org> on 2014/03/16 06:59:24 UTC
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
GitHub user gzm55 opened a pull request:
https://github.com/apache/spark/pull/154
remove staging dir when app quiting for yarn-cluster mode
In yarn-cluster, the driver is actually running as 'yarn' user. When
posting jobs from other users, we need give stagingDir a full path, so
the driver could remove the correct directory if needed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gzm55/spark work/rm-yarn-staging-dir
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/154.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #154
----
commit eaad42d434792639424292fdc17f07eba5d3646e
Author: James Z.M. Gao <ga...@mediav.com>
Date: 2014-03-11T03:07:28Z
remove staging dir when app quiting for yarn-cluster mode
In yarn-cluster, the driver is actually running as 'yarn' user. When
posting jobs from other users, we need give stagingDir a full path, so
the driver could remove the correct directory if needed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-66519309
Hey @gzm55 mind closing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/154
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-58241932
Hey @gzm55 are you still working on this? Could you add `[Yarn]` and an associated JIRA to the title? See how other PRs are opened.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by gzm55 <gi...@git.apache.org>.
Github user gzm55 commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-49120476
@tgravescs I file a jira describing this bug: https://issues.apache.org/jira/browse/SPARK-2508
In a yarn-cluster, the driver is actually running as 'yarn' user in some container by a nodemanager. When
submitting jobs from other users, e.g. user1, the stagingDir is created in submitting stage, which is actually a progress run by user1. So the full path of stagingDir on the hdfs should look like '/user/user1/.staging/...'.
And in cleaning up stage, stagingDir could be removed by the driver. At this time, we need give stagingDir a full path, so
the driver could remove the correct directory ('/user/user1/.staging/...'), not '/user/yarn/.staging/...'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-61989912
@gzm55 are you still working on this, if not can we close it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-64868760
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by mridulm <gi...@git.apache.org>.
Github user mridulm commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-37751154
jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-37750476
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-62561980
ping @gzm55. Any chance you can update this?
If not can we close this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-45634494
@gzm55 Is this still an issue with spark 1.0 release?
If so can you file a jira and provide more details on your hadoop/yarn setup? Are you running your yarn cluster daemons with user "yarn" and then also letting the jobs run as user "yarn"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/154#discussion_r17480000
--- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala ---
@@ -273,7 +273,7 @@ trait ClientBase extends Logging {
ClientBase.populateClasspath(yarnConf, sparkConf, log4jConfLocalRes != null, env)
env("SPARK_YARN_MODE") = "true"
- env("SPARK_YARN_STAGING_DIR") = stagingDir
+ env("SPARK_YARN_STAGING_DIR") = FileSystem.get(conf).getHomeDirectory() + Path.SEPARATOR + stagingDir
--- End diff --
Since this function already takes the stagingDir as a parameter and since we have to construct this now in 2 places (here and in prepareLocalResources) it would be better to pull this up higher and construct it once and pass it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-54694755
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-55407631
Note that this pr will also probably conflict with https://github.com/apache/spark/pull/2350
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: remove staging dir when app quiting for yarn-c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/154#issuecomment-38881881
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---