You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2015/01/09 11:22:34 UTC
[jira] [Updated] (SPARK-5172) spark-examples-***.jar shades a wrong
Hadoop distribution
[ https://issues.apache.org/jira/browse/SPARK-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shixiong Zhu updated SPARK-5172:
--------------------------------
Component/s: Build
Priority: Minor (was: Major)
> spark-examples-***.jar shades a wrong Hadoop distribution
> ---------------------------------------------------------
>
> Key: SPARK-5172
> URL: https://issues.apache.org/jira/browse/SPARK-5172
> Project: Spark
> Issue Type: Bug
> Components: Build
> Reporter: Shixiong Zhu
> Priority: Minor
>
> Steps to check it:
> 1. Download "spark-1.2.0-bin-hadoop2.4.tgz" from http://www.apache.org/dyn/closer.cgi/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
> 2. unzip `spark-assembly-1.2.0-hadoop2.4.0.jar`.
> 3. There is a file called `org/apache/hadoop/package-info.class` in the jar. It doesn't exist in hadoop 2.4.
> 4. Run "javap -classpath . -private -c -v org.apache.hadoop.package-info"
> {code}
> Compiled from "package-info.java"
> interface org.apache.hadoop.package-info
> SourceFile: "package-info.java"
> RuntimeVisibleAnnotations: length = 0x24
> 00 01 00 06 00 06 00 07 73 00 08 00 09 73 00 0A
> 00 0B 73 00 0C 00 0D 73 00 0E 00 0F 73 00 10 00
> 11 73 00 12
> minor version: 0
> major version: 50
> Constant pool:
> const #1 = Asciz org/apache/hadoop/package-info;
> const #2 = class #1; // "org/apache/hadoop/package-info"
> const #3 = Asciz java/lang/Object;
> const #4 = class #3; // java/lang/Object
> const #5 = Asciz package-info.java;
> const #6 = Asciz Lorg/apache/hadoop/HadoopVersionAnnotation;;
> const #7 = Asciz version;
> const #8 = Asciz 1.2.1;
> const #9 = Asciz revision;
> const #10 = Asciz 1503152;
> const #11 = Asciz user;
> const #12 = Asciz mattf;
> const #13 = Asciz date;
> const #14 = Asciz Wed Jul 24 13:39:35 PDT 2013;
> const #15 = Asciz url;
> const #16 = Asciz https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2;
> const #17 = Asciz srcChecksum;
> const #18 = Asciz 6923c86528809c4e7e6f493b6b413a9a;
> const #19 = Asciz SourceFile;
> const #20 = Asciz RuntimeVisibleAnnotations;
> {
> }
> {code}
> The version is {{1.2.1}}
> It comes because a wrong hbase version settings in examples project. Here is a port of the dependencly tree when runnning "mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples dependency:tree"
> {noformat}
> [INFO] +- org.apache.hbase:hbase-testing-util:jar:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hbase:hbase-common:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hbase:hbase-server:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | | +- com.sun.jersey:jersey-core:jar:1.8:compile
> [INFO] | | +- com.sun.jersey:jersey-json:jar:1.8:compile
> [INFO] | | | +- org.codehaus.jettison:jettison:jar:1.1:compile
> [INFO] | | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
> [INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.7.1:compile
> [INFO] | | \- com.sun.jersey:jersey-server:jar:1.8:compile
> [INFO] | | \- asm:asm:jar:3.3.1:test
> [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hbase:hbase-hadoop1-compat:test-jar:tests:0.98.7-hadoop1:compile
> [INFO] | +- org.apache.hadoop:hadoop-core:jar:1.2.1:compile
> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> [INFO] | | \- commons-el:commons-el:jar:1.0:compile
> [INFO] | +- org.apache.hadoop:hadoop-test:jar:1.2.1:compile
> [INFO] | | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:compile
> [INFO] | | +- org.apache.mina:mina-core:jar:2.0.0-M5:compile
> [INFO] | | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:compile
> [INFO] | | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:compile
> [INFO] | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
> [INFO] | \- junit:junit:jar:4.10:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.1:test
> {noformat}
> If I ran `mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -pl examples -am dependency:tree -Dhbase.profile=hadoop2`, the dependency tree is right.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org