You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2018/05/21 00:21:00 UTC
[jira] [Comment Edited] (YARN-8079) Support static and archive
unmodified local resources in service AM
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482076#comment-16482076 ]
Eric Yang edited comment on YARN-8079 at 5/21/18 12:20 AM:
-----------------------------------------------------------
[~leftnoteasy] When the files are placed in resources directory, patch 10 implementation prevents mistake to overwrite system level generated files, such as .token file, and launch_container.sh. However, this design can created inconvenience for some users because existing Hadoop workload may already be using the top level localized directory instead of resource directory. We may not need to worry about launch_container.sh getting overwritten because container-executor generates the file after static files are localized. Apps will try to avoid .token files because they can not contact HDFS from containers, if they overwrites the token files.
With resources directory, it maybe easier for end user to specify a single relative directory to bind-mount instead of specifying individual files to bind-mount in yarnfile. By removing the resources directory, user will need to think a bit more on how to manage the bind-mount directories to reduce wordy syntax.
With both approaches considered, it all comes down to usability of which approach is easiest to use, while not creating too much clutters. In summary, it might be safe to remove the requirement of "resources" directory from my point of view.
was (Author: eyang):
[~leftnoteasy] When the files are placed in resources directory, patch 10 implementation prevents mistake to overwrite system level generated files, such as .token file, and launch_container.sh. However, this design can created inconvenience for some users because existing Hadoop workload may already be using the top level localized directory instead of resource directory. We may not need to worry about launch_container.sh getting overwritten because container-executor generates the file after static files are localized. Apps will try to avoid .token files because they can not contact HDFS from containers, if they overwrites the token files. In summary, it is likely safe to remove the requirement of "resources" directory from my point of view.
> Support static and archive unmodified local resources in service AM
> -------------------------------------------------------------------
>
> Key: YARN-8079
> URL: https://issues.apache.org/jira/browse/YARN-8079
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Suma Shivaprasad
> Priority: Critical
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8079.001.patch, YARN-8079.002.patch, YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, YARN-8079.009.patch, YARN-8079.010.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly read srcFile, instead it always construct {{remoteFile}} by using componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code}
> To me it is a common use case which services have some files existed in HDFS and need to be localized when components get launched. (For example, if we want to serve a Tensorflow model, we need to localize Tensorflow model (typically not huge, less than GB) to local disk. Otherwise launched docker container has to access HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org