You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mingyu Kim (JIRA)" <ji...@apache.org> on 2015/03/31 03:27:52 UTC
[jira] [Updated] (SPARK-6619) Jar cache on Executors should use
file content hash as the key
[ https://issues.apache.org/jira/browse/SPARK-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mingyu Kim updated SPARK-6619:
------------------------------
Description:
Taking SPARK-2713 one step further so that the cached jars can be used by multiple applications. In order to do that, I'm planning to use MD5 as the cache key as opposed to url hash and timestamp.
Re: perf. Computing MD5 using "openssl" on my local Macbook Pro took 1.2s for 158 jars with the total size of 56MB, and this takes 5~10s to
was:Taking SPARK-2713 one step further so that the cached jars can be used by multiple applications. In order to do that, I'm planning to use MD5 as the cache key as opposed to url hash and timestamp.
> Jar cache on Executors should use file content hash as the key
> --------------------------------------------------------------
>
> Key: SPARK-6619
> URL: https://issues.apache.org/jira/browse/SPARK-6619
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Mingyu Kim
>
> Taking SPARK-2713 one step further so that the cached jars can be used by multiple applications. In order to do that, I'm planning to use MD5 as the cache key as opposed to url hash and timestamp.
> Re: perf. Computing MD5 using "openssl" on my local Macbook Pro took 1.2s for 158 jars with the total size of 56MB, and this takes 5~10s to
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org