[jira] [Created] (YARN-5396) YARN large file broadcast service

["Zhiyuan Yang (JIRA)" <ji...@apache.org>]

Zhiyuan Yang created YARN-5396:
----------------------------------

             Summary: YARN large file broadcast service
                 Key: YARN-5396
                 URL: https://issues.apache.org/jira/browse/YARN-5396
             Project: Hadoop YARN
          Issue Type: New Feature
            Reporter: Zhiyuan Yang


In Hadoop and related softwares, there are demands of broadcasting large files. For example, YARN application may localize large jar files on each node; Hive may distribute large tables in fragment-replicate joins; docker integration may broadcast large container image. The current local resource based solution is to put the files on HDFS and let each node download from HDFS, which is inefficient and not scalable. So we want to build a better file transfer service in YARN so that all applications can use it broadcast large file efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org