You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "James Z.M. Gao (JIRA)" <ji...@apache.org> on 2014/07/16 05:38:04 UTC

[jira] [Created] (SPARK-2508) fail to remove staging dir when app quits in a yarn-cluster

James Z.M. Gao created SPARK-2508:
-------------------------------------

             Summary: fail to remove staging dir when app quits in a yarn-cluster
                 Key: SPARK-2508
                 URL: https://issues.apache.org/jira/browse/SPARK-2508
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.0.0, 0.9.1, 0.9.0
         Environment: RedHat 5.3
2.0.0-cdh4.6.0
java version "1.6.0_45"
            Reporter: James Z.M. Gao
            Priority: Minor


Hi,

In a yarn-cluster, the driver is actually running as 'yarn' user in some container by a nodemanager. When
submitting jobs from other users, e.g. user1, the stagingDir is created in submitting stage, which is actually a progress run by user1. So the full path of stagingDir on the hdfs should look like '/user/user1/.staging/...'.

And in cleaning up stage, stagingDir could be removed by the driver. At this time, we need give stagingDir a full path, so
the driver could remove the correct directory ('/user/user1/.staging/...'), not '/user/yarn/.staging/...'.

Here is a simple PR: https://github.com/apache/spark/pull/154



--
This message was sent by Atlassian JIRA
(v6.2#6252)