[jira] [Comment Edited] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources

["Jian He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475973#comment-15475973 ] 

Jian He edited comment on YARN-5621 at 9/9/16 5:32 AM:
-------------------------------------------------------

bq. the container launch script is pretty self contained, is mostly controlled by the NM, and there are other actions in the pipeline. Running a generic script without any of that extra baggage around it seems to be greatly expanding the footprint of c-e.
I still don't see the difference. Both are controlled by NM. The launch_container.sh can run any arbitrary user supplied command. If you just pass dummy objects to other parameters of container launch command, it's essentially doing exactly the same thing.
bq. Why can't the directory structure be built by the NM and it just be a chown operation by c-e? 
I guess the question is why the original container_launch script is not done in this way? Even if it's done this way, should we then add a new 'chown' API in the container-executor ? or should it belong to the API of creating symlinks (similar as a script)?   It's much easier to follow existing container_launch code which takes care of all environments, rather than inventing new. Also, later on we need to create multiple symlinks in a single operation as done in current container_launch script, because if there is a large number of local Resources to be localized, we don't want to invoke the binary for each of them.  




was (Author: jianhe):
bq. the container launch script is pretty self contained, is mostly controlled by the NM, and there are other actions in the pipeline. Running a generic script without any of that extra baggage around it seems to be greatly expanding the footprint of c-e.
I still don't see the difference. Both are controlled by NM. The launch_container.sh runs any arbitrary user supplied command and this feature does the same thing. If you just pass dummy objects to other parameters of container launch command, it's essentially doing exactly the same thing.
bq. Why can't the directory structure be built by the NM and it just be a chown operation by c-e? 
I guess the question is why the original container_launch script is not done in this way? Even if it's done this way, should we then add a new 'chown' API in the container-executor ? or should it belong to the API of creating symlinks (similar as a script)?   It's much easier to follow existing container_launch code which takes care of all environments, rather than inventing new. Also, later on we need to create multiple symlinks in a single operation as done in current container_launch script, because if there is a large number of local Resources to be localized, we don't want to invoke the binary for each of them.  



> Support LinuxContainerExecutor to create symlinks for continuously localized resources
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch
>
>
> When new resources are localized, new symlink needs to be created for the localized resource. This is the change for the LinuxContainerExecutor to create the symlinks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org