[jira] [Commented] (FLINK-8279) Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308077#comment-16308077 ] 

ASF GitHub Bot commented on FLINK-8279:
---------------------------------------

Github user NicoK commented on the issue:

    https://github.com/apache/flink/pull/5176
  
    @EronWright this PR originated from a user problem at https://lists.apache.org/thread.html/177644ad38af34fb7ae7fb12a3bfcef3a70cd70dfccb6f7008a24eb6@%3Cuser.flink.apache.org%3E
    
    And yes, this also adds the fallback to (a subdirectory in) `taskmanager.tmp.dirs` in case no `blob.storage.directory` was given to the JobManager. Given that the configuration parameter hints towards the TaskManager only, we should maybe rename it?
    
    Regarding [MAPREDUCE-6472](https://issues.apache.org/jira/browse/MAPREDUCE-6472), it seems this is only available from Hadoop 2.7.2 onwards. With this fix applied, the behaviour changes only slightly: instead of using `java.io.tmpdir` directly (set to `Environment.PWD/tmp`), this will use the `taskmanager.tmp.dirs` which is set to `Environment.LOCAL_DIRS` by `YarnTaskManagerRunner`/`YarnTaskExecutorRunner`.
    
    I adapted the appropriate code paths for Mesos as well. If this is not used yet, it will at least work in the future once the `taskmanager.tmp.dirs` handling is correct. Until then, the fallback of this to `java.io.tmpdir` seems to be working as well (if this is set appropriately by Mesos) - but this is not only a problem of the BLOB server parts then and is out of the scope of this PR.


> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-8279
>                 URL: https://issues.apache.org/jira/browse/FLINK-8279
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination, Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming files into their local file system in the directory given by the {{blob.storage.directory}} configuration property. If this property is not set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories they assigned to the Flink job which are not only the proper folder to use, but may also offer some more space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)