You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2016/04/27 20:17:13 UTC

[jira] [Created] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

Thomas Graves created SPARK-14963:
-------------------------------------

             Summary: YarnShuffleService should use YARN getRecoveryPath() for leveldb location
                 Key: SPARK-14963
                 URL: https://issues.apache.org/jira/browse/SPARK-14963
             Project: Spark
          Issue Type: Bug
          Components: Shuffle, YARN
    Affects Versions: 1.6.1
            Reporter: Thomas Graves


The YarnShuffleService, currently just picks a directly in the yarn local dirs to store the leveldb file.  YARN added an interface in hadoop 2.5 getRecoverPath() to get the location where it should be storing this.

We should change to use getRecoveryPath(). This does mean we will have to use reflection or similar to check for its existence though since it doesn't exist before hadoop 2.5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org