You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Harry Brundage (JIRA)" <ji...@apache.org> on 2015/06/17 17:23:01 UTC
[jira] [Comment Edited] (SPARK-7009) Build assembly JAR via ant to
avoid zip64 problems
[ https://issues.apache.org/jira/browse/SPARK-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589913#comment-14589913 ]
Harry Brundage edited comment on SPARK-7009 at 6/17/15 3:22 PM:
----------------------------------------------------------------
[~sowen] I wrote a script to test that we can all run to prove if it's importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1 hadoop-2.6 artifact works fine for me on my machine. Script attached.
If this is the case for others, then we're in a spot of trouble, no?
was (Author: airhorns):
[~sowen] I wrote a script to test that we can all run to prove if it's importable or not and check my assumptions. Both the 1.4 hadoop-2.6 and hadoop-provided artifacts fail to be imported by python, whereas the 1.3.1 hadoop-2.6 artifact works fine for me on my machine. Script attached.
If this is the case for other, then we're in a spot of trouble, no?
> Build assembly JAR via ant to avoid zip64 problems
> --------------------------------------------------
>
> Key: SPARK-7009
> URL: https://issues.apache.org/jira/browse/SPARK-7009
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.0
> Environment: Java 7+
> Reporter: Steve Loughran
> Attachments: check_spark_python.sh
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> SPARK-1911 shows the problem that JDK7+ is using zip64 to build large JARs; a format incompatible with Java and pyspark.
> Provided the total number of .class files+resources is <64K, ant can be used to make the final JAR instead, perhaps by unzipping the maven-generated JAR then rezipping it with zip64=never, before publishing the artifact via maven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org