You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Wangda Tan (JIRA)" <ji...@apache.org> on 2018/12/03 18:06:00 UTC
[jira] [Commented] (YARN-8714) [Submarine] Support files/tarballs
to be localized for a training job.
[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707602#comment-16707602 ]
Wangda Tan commented on YARN-8714:
----------------------------------
[~liuxun323], fair enough.
[~tangzhankun], I think we can add a setting to submarine config. By default set it to 2GB or so. And print logs when we download files, tar it locally and upload to HDFS to make troubleshooting easier. Also please remove the local tmp file once upload is done. Another concern is if this operation needs to be done repeatedly for every submitted job, it gonna be a big issue. If we could append directory's modification time and size to the tar file for now, later we can optimize it to share same uploaded files across jobs.
> [Submarine] Support files/tarballs to be localized for a training job.
> ----------------------------------------------------------------------
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Zhankun Tang
> Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch, YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch, YARN-8714-trunk.002.patch
>
>
> See [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7], {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org