[jira] [Resolved] (SPARK-756) JAR file appears corrupt to workers. - MLbase

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-756.
-----------------------------
    Resolution: Cannot Reproduce

> JAR file appears corrupt to workers. - MLbase
> ---------------------------------------------
>
>                 Key: SPARK-756
>                 URL: https://issues.apache.org/jira/browse/SPARK-756
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.8.0
>            Reporter: Evan Sparks
>
> Running Spark on EC2 with the default AMI and standard configuration.
> This bug affects the spark master branch (as of 2013-05-29) - notably, the same code does not crash under spark 0.7.0.
> I am creating a 55MB JAR file via the sbt-assembly plugin which is valid - code inside the JAR is callable and works fine if I rsync it to each worker and put that file  on the SPARK_CLASSPATH. 
> However, when I share the JAR by passing it to the SparkContext (e.g. : val sc = new SparkContext("spark://"+ sys.env("SPARK_MASTER_IP") + ":7077", "VisionClass", "/root/spark", List("target/visionClass-assembly-1.0.jar")) )
> Worker tasks die - with a "java.util.zip.ZipException" when attempting to load the JAR file.
> The jar file in the spark worker temp directory seems complete (MD5 sums match what I have on the master) upon later inspection - so I suspect the Worker is attempting to open the file before it has been fully flushed to disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org