You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Haibo Chen (JIRA)" <ji...@apache.org> on 2018/12/06 23:18:00 UTC
[jira] [Commented] (YARN-9008) Extend YARN distributed shell with
file localization feature
[ https://issues.apache.org/jira/browse/YARN-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712143#comment-16712143 ]
Haibo Chen commented on YARN-9008:
----------------------------------
Thanks [~pbacsko] for the patch! A few minor comments
1) We are missing one unit test for upload a non-existent file and one for a directory.
2) The new commandline option 'appname' should probably be renamed to 'app_name' for the sake of consistency with other options
3) All IOExceptions are wrapped in a RunTimeException. But I am not sure why benefits it provides than just directly throwing IOException.
4) I notice 2.9.1 is included in the affect version. Do you intend to backport this into branch-2? If so, we shall not use stream api that is only supported in Java 8.
5) The relative path of a file is composed of the app_name, appId and the file name. We have two copies of the same code in both ApplicationMaster and Client. If only one copy is changed in the future, the feature would fail. Can we centralize them in one place?
6) 'localized_files' sounds very much into the implementation details. MapReduce jobs client can add lib files at submission time, which are under the hood uploaded to HDFS and localized for access. We have almost the same idea here. What do you think of renaming it to 'lib'?
> Extend YARN distributed shell with file localization feature
> ------------------------------------------------------------
>
> Key: YARN-9008
> URL: https://issues.apache.org/jira/browse/YARN-9008
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 2.9.1, 3.1.1
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Priority: Major
> Attachments: YARN-9008-001.patch, YARN-9008-002.patch, YARN-9008-003.patch, YARN-9008-004.patch
>
>
> YARN distributed shell is a very handy tool to test various features of YARN.
> However, it lacks support for file localization - that is, you define files in the command line that you wish to be localized remotely. This can be extremely useful in certain scenarios.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org