[jira] [Commented] (FLINK-9762) CoreOptions.TMP_DIRS wrongly managed on Yarn

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16536737#comment-16536737 ] 

ASF GitHub Bot commented on FLINK-9762:
---------------------------------------

GitHub user JTaky opened a pull request:

    https://github.com/apache/flink/pull/6284

    [FLINK-9762] CoreOptions.TMP_DIRS ('io.tmp.dirs') wrongly managed on Yarn

    ## What is the purpose of the change
    
    Currently TMP_DIRS (io.tmp.dirs) option has a bug in case of default behaviour on Yarn platform:
    If not value has been set via parameter on Yarn platform the env value 'LOCAL_DIRS' is used on Application Master.
    When task manager is started this 'LOCAL_DIRS' is passed to the TaskManager, so TaskManager do not understand that he needs to have a default behaviour.
    The proposed ~~hack~~ fix is to remove TMP_DIRS value in case if default behaviour is required before starting TaskManager.
    
    ## Brief change log
    
    * Extract method which set _io.tmp.dirs_ configuration value
    * Fix configuration pass to TaskManager (**I am not sure what is the best way to implement this**):
    ** Add a new flag, saying _io.tmp.dirs_ is value has been overridden
    ** If _io.tmp.dirs_ has *not* been overridden we remove _io.tmp.dirs_ remove config value, by setting the empty value (should we remove? clone without a given key?)
    * Correction of the documentation
    
    ## Verifying this change
    
    This change is already covered by existing tests, such as MiniClusterResource for Mesos part.
    1. I do not see a way to test launching of the TaskManager with a correct configuration.
    2. A unit test documenting specific logic of TMP_DIRS managing is added: BootstrapToolsTest#testSetTmpDirectoriesConfig
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): no
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: no
      - The runtime per-record code paths (performance sensitive): no
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
      - The S3 file system connector: no
    
    ## Documentation
    
      - Does this pull request introduce a new feature? no


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/JTaky/flink tmpDirs

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/6284.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6284
    
----
commit d0c0bf7682843947bd1cfdf27775eb75c2864232
Author: Oleksandr Nitavskyi <o....@...>
Date:   2018-07-02T07:42:17Z

    [FLINK-9762][yarn] Extract method: tmp dirs setting

commit 99bdeb924a2bf91eab39462ac8253ee79dcb1137
Author: Oleksandr Nitavskyi <o....@...>
Date:   2018-07-02T08:06:56Z

    [FLINK-9762][yarn] Fix TMP_DIRS configuration passing
    
    * HACK: Instead of removing set value as empty

commit 2fd5754209508e082e4678e8605a25e3d25f8209
Author: Oleksandr Nitavskyi <o....@...>
Date:   2018-07-09T09:15:14Z

    [FLINK-9762] correct documenatation for 'io.tmp.dirs' config

----


> CoreOptions.TMP_DIRS wrongly managed on Yarn
> --------------------------------------------
>
>                 Key: FLINK-9762
>                 URL: https://issues.apache.org/jira/browse/FLINK-9762
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.5.0
>            Reporter: Oleksandr Nitavskyi
>            Assignee: vinoyang
>            Priority: Major
>              Labels: pull-request-available
>
> The issue on Yarn is that it is impossible to have different LOCAL_DIRS on JobManager and TaskManager, despite LOCAL_DIRS value depends on the container.
> The issue is that CoreOptions.TMP_DIRS is configured to the default value during JobManager initialization and added to the configuration object. When TaskManager is launched the appropriate configuration object is cloned with LOCAL_DIRS which makes sense only for Job Manager container. When YARN container with TaskManager from his point of view CoreOptions.TMP_DIRS is always equal either to path in flink.yml or to the or to the LOCAL_DIRS of Job Manager (default behaviour). Is TaskManager’s container do not have an access to another folders, that folders allocated by YARN TaskManager cannot be started.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)