[jira] [Closed] (SPARK-2713) Executors of same application in same host should only download files & jars once

["Andrew Or (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Or closed SPARK-2713.
----------------------------
          Resolution: Fixed
       Fix Version/s: 1.2.0
            Assignee: Zhihui
    Target Version/s: 1.2.0

> Executors of same application in same host should only download files & jars once
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-2713
>                 URL: https://issues.apache.org/jira/browse/SPARK-2713
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Zhihui
>            Assignee: Zhihui
>             Fix For: 1.2.0
>
>
> If Spark lunched multiple executors in one host for one application, every executor would download it dependent files and jars (if not using local: url) independently. It maybe result in huge latency. In my case, it result in 20 seconds latency to download dependent jars(about 17M) when I lunch 32 executors in one host(total 4 hosts). 
> This patch will cache downloaded files and jars for executors to reduce network throughput and download latency. I my case, the latency was reduced from 20 seconds to less than 1 second.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org