You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ey-chih chow <ey...@hotmail.com> on 2015/01/22 01:51:21 UTC
spark 1.1.0 save data to hdfs failed
Hi,
I used the following fragment of a scala program to save data to hdfs:
contextAwareEvents
.map(e => (new AvroKey(e), null))
.saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
+ dateDir,
classOf[AvroKey[GenericRecord]],
classOf[NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
job.getConfiguration)
But it failed with the following error messages. Is there any people who
can help? Thanks.
Ey-Chih Chow
=============================================
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.io.IOException: Failed on local exception:
java.io.EOFException; Host Details : local host is:
"ip-10-33-140-157/10.33.140.157"; destination host is:
"ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
... 6 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
=======================================================
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
After I changed the dependency to the following:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.2.1</version>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
I got the following error. Any idea on this? Thanks.
===============Caused by: java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
at org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
at org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
... 6 moreFrom: eychih@hotmail.com
To: sowen@cloudera.com
CC: yuzhihong@gmail.com; user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Thu, 22 Jan 2015 17:05:26 -0800
Thanks. But after I replace the maven dependence from
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.5.0-cdh5.2.0</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>servlet-api</artifactId> </exclusion> <exclusion> <groupId>javax.servlet</groupId> <artifactId>servlet-api</artifactId> </exclusion> <exclusion> <groupId>io.netty</groupId> <artifactId>netty</artifactId> </exclusion> </exclusions> </dependency>
to <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.0.4</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
the warning message is still shown up in the namenode log. Is there any other thing I need to do?
Thanks.
Ey-Chih Chow
> From: sowen@cloudera.com
> Date: Thu, 22 Jan 2015 22:34:22 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: yuzhihong@gmail.com; user@spark.apache.org
>
> It means your client app is using Hadoop 2.x and your HDFS is Hadoop 1.x.
>
> On Thu, Jan 22, 2015 at 10:32 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > I looked into the namenode log and found this message:
> >
> > 2015-01-22 22:18:39,441 WARN org.apache.hadoop.ipc.Server: Incorrect header
> > or version mismatch from 10.33.140.233:53776 got version 9 expected version
> > 4
> >
> > What should I do to fix this?
> >
> > Thanks.
> >
> > Ey-Chih
> >
> > ________________________________
> > From: eychih@hotmail.com
> > To: yuzhihong@gmail.com
> > CC: user@spark.apache.org
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > Date: Wed, 21 Jan 2015 23:12:56 -0800
> >
> > The hdfs release should be hadoop 1.0.4.
> >
> > Ey-Chih Chow
> >
> > ________________________________
> > Date: Wed, 21 Jan 2015 16:56:25 -0800
> > Subject: Re: spark 1.1.0 save data to hdfs failed
> > From: yuzhihong@gmail.com
> > To: eychih@hotmail.com
> > CC: user@spark.apache.org
> >
> > What hdfs release are you using ?
> >
> > Can you check namenode log around time of error below to see if there is
> > some clue ?
> >
> > Cheers
> >
> > On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
> >
> > Hi,
> >
> > I used the following fragment of a scala program to save data to hdfs:
> >
> > contextAwareEvents
> > .map(e => (new AvroKey(e), null))
> > .saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
> > + dateDir,
> > classOf[AvroKey[GenericRecord]],
> > classOf[NullWritable],
> > classOf[AvroKeyOutputFormat[GenericRecord]],
> > job.getConfiguration)
> >
> > But it failed with the following error messages. Is there any people who
> > can help? Thanks.
> >
> > Ey-Chih Chow
> >
> > =============================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.io.IOException: Failed on local exception:
> > java.io.EOFException; Host Details : local host is:
> > "ip-10-33-140-157/10.33.140.157"; destination host is:
> > "ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1415)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> > at
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> > at
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> > at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
> > at
> > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> > at
> > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> > ... 6 more
> > Caused by: java.io.EOFException
> > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > at
> > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
> >
> > =======================================================
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
I also think the code is not robust enough. First, Spark works with hadoop1, why the code try hadoop2 first. Also the following code only handle ClassNotFoundException. It should handle all the exceptions.
private def firstAvailableClass(first: String, second: String): Class[_] = { try { Class.forName(first) } catch { case e: ClassNotFoundException => Class.forName(second) } }
From: eychih@hotmail.com
To: sowen@cloudera.com
CC: user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Fri, 23 Jan 2015 06:43:00 -0800
I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it is broken in the following code:
def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID): TaskAttemptContext = { val klass = firstAvailableClass( "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", // hadoop2, hadoop2-yarn "org.apache.hadoop.mapreduce.TaskAttemptContext") // hadoop1 val ctor = klass.getDeclaredConstructor(classOf[Configuration], classOf[TaskAttemptID]) ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext] }
In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any suggestion how to resolve it?
Thanks.
> From: sowen@cloudera.com
> Date: Fri, 23 Jan 2015 14:01:45 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> These are all definitely symptoms of mixing incompatible versions of libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the dependency tree.
>
> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> > provided. Why the cluster's version is not used?
> >
> > Any way, as I mentioned in the previous message, after changing the
> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> > exception and go to another one as indicated below. Any suggestion on this?
> >
> > =================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:191)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >
> > ... 6 more
> >
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
I modified my pom.xml according to the Spark pom.xml. It is working right now. Hadoop2 classes are no longer packaged into my jar. Thanks.
From: eychih@hotmail.com
To: sowen@cloudera.com
CC: user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Sat, 24 Jan 2015 07:30:45 -0800
Thanks for the information. I changed the dependencies for Spark jars as follows:
<dependency> <!-- Spark -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark SQL -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>
I don't know how these libraries are built, but I saw Spark has maven pom files. I think these jars should be built from the corresponding pom files. These pom files have dependencies on hadoop version 1.0.4. So I don't know where the hadoop2 jar come from. What follows is a major fragment of my current dependency tree. I don't know where the hadoop2 classes come into my built jar.
======================
[INFO] | \- org.apache.hadoop:hadoop-core:jar:1.2.1:provided
[INFO] | +- xmlenc:xmlenc:jar:0.52:provided
[INFO] | +- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (com.sun.jersey:jersey-json:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (com.sun.jersey:jersey-server:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (commons-io:commons-io:jar:2.1:provided - omitted for conflict with 2.4)
[INFO] | +- (commons-codec:commons-codec:jar:1.4:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- (org.apache.commons:commons-math:jar:2.1:provided - omitted for duplicate)
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided
[INFO] | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | +- (commons-lang:commons-lang:jar:2.4:provided - omitted for conflict with 2.6)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- commons-digester:commons-digester:jar:1.8:provided
[INFO] | | | +- commons-beanutils:commons-beanutils:jar:1.7.0:provided
[INFO] | | | | \- (commons-logging:commons-logging:jar:1.0.3:provided - omitted for conflict with 1.1.1)
[INFO] | | | \- (commons-logging:commons-logging:jar:1.1:provided - omitted for conflict with 1.1.1)
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:provided
[INFO] | | \- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | +- (commons-net:commons-net:jar:1.4.1:provided - omitted for conflict with 2.2)
[INFO] | +- commons-el:commons-el:jar:1.0:provided
[INFO] | | \- (commons-logging:commons-logging:jar:1.0.3:provided - omitted for conflict with 1.1.1)
[INFO] | +- hsqldb:hsqldb:jar:1.8.0.10:provided
[INFO] | +- oro:oro:jar:2.0.8:provided
[INFO] | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.1.0:provided
[INFO] | +- (org.apache.hadoop:hadoop-client:jar:1.0.4:provided - omitted for conflict with 1.2.1)
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:provided
[INFO] | | +- (commons-codec:commons-codec:jar:1.3:provided - omitted for conflict with 1.4)
[INFO] | | \- (commons-httpclient:commons-httpclient:jar:3.1:provided - omitted for duplicate)
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:provided
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:provided
[INFO] | | | +- org.apache.curator:curator-client:jar:2.4.0:provided
[INFO] | | | | +- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.6.1)
[INFO] | | | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for duplicate)
[INFO] | | | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for duplicate)
[INFO] | | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for conflict with 3.4.6)
[INFO] | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | +- org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031:provided
[INFO] | | +- org.eclipse.jetty.orbit:javax.transaction:jar:1.1.1.v201105210645:provided
[INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:8.1.14.v20131031:provided
[INFO] | | | +- org.eclipse.jetty:jetty-xml:jar:8.1.14.v20131031:provided
[INFO] | | | | \- (org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:provided
[INFO] | | | \- (org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | \- org.eclipse.jetty:jetty-jndi:jar:8.1.14.v20131031:provided
[INFO] | | +- (org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | \- org.eclipse.jetty.orbit:javax.mail.glassfish:jar:1.4.1.v201005082020:provided
[INFO] | | \- org.eclipse.jetty.orbit:javax.activation:jar:1.1.0.v201105071233:provided
[INFO] | +- org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:provided
[INFO] | | \- (org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | +- org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided
[INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided
[INFO] | | +- org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:provided
[INFO] | | +- org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:provided
[INFO] | | \- org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:provided
[INFO] | | \- org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:provided
[INFO] | | \- (org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | +- (com.google.guava:guava:jar:14.0.1:provided - omitted for conflict with 14.0)
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:provided
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:provided
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.5:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.5:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.5:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- log4j:log4j:jar:1.2.17:provided
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | | \- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | +- com.ning:compress-lzf:jar:1.0.0:provided
[INFO] | +- (org.xerial.snappy:snappy-java:jar:1.0.5.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- net.jpountz.lz4:lz4:jar:1.2.0:provided
[INFO] | +- com.twitter:chill_2.10:jar:0.3.6:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.3:provided - omitted for conflict with 2.10.4)
[INFO] | | +- (com.twitter:chill-java:jar:0.3.6:provided - omitted for duplicate)
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:provided
[INFO] | | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:provided
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:provided
[INFO] | | \- org.objenesis:objenesis:jar:1.2:provided
[INFO] | +- com.twitter:chill-java:jar:0.3.6:provided
[INFO] | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:provided - omitted for duplicate)
[INFO] | +- commons-net:commons-net:jar:2.2:provided
[INFO] | +- org.spark-project.akka:akka-remote_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | +- org.spark-project.akka:akka-actor_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | | \- com.typesafe:config:jar:1.0.2:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | +- (io.netty:netty:jar:3.6.6.Final:provided - omitted for duplicate)
[INFO] | | +- org.spark-project.protobuf:protobuf-java:jar:2.4.1-shaded:provided
[INFO] | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:provided
[INFO] | +- org.spark-project.akka:akka-slf4j_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | +- (org.spark-project.akka:akka-actor_2.10:jar:2.2.3-shaded-protobuf:provided - omitted for duplicate)
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | +- org.scala-lang:scala-library:jar:2.10.4:provided
[INFO] | +- org.json4s:json4s-jackson_2.10:jar:3.2.10:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | +- org.json4s:json4s-core_2.10:jar:3.2.10:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:provided
[INFO] | | | | \- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | | +- (com.thoughtworks.paranamer:paranamer:jar:2.6:provided - omitted for conflict with 2.3)
[INFO] | | | \- org.scala-lang:scalap:jar:2.10.0:provided
[INFO] | | | \- (org.scala-lang:scala-compiler:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | \- (com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:provided - omitted for conflict with 2.3.0)
[INFO] | +- colt:colt:jar:1.2.0:provided
[INFO] | | \- concurrent:concurrent:jar:1.3.4:provided
[INFO] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1:provided
[INFO] | +- io.netty:netty-all:jar:4.0.23.Final:provided
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] | +- com.codahale.metrics:metrics-core:jar:3.0.0:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-jvm:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-json:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.2:provided - omitted for conflict with 2.3.1)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-graphite:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-jvm:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-json:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.2:provided - omitted for conflict with 2.3.1)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-graphite:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- org.tachyonproject:tachyon-client:jar:0.5.0:provided
[INFO] | | \- org.tachyonproject:tachyon:jar:0.5.0:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (org.slf4j:slf4j-log4j12:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | +- (commons-io:commons-io:jar:2.4:provided - omitted for conflict with 2.1)
[INFO] | | +- (org.apache.commons:commons-lang3:jar:3.0:provided - omitted for conflict with 3.3.2)
[INFO] | | \- (com.fasterxml.jackson.core:jackson-databind:jar:2.3.0:provided - omitted for conflict with 2.3.1)
[INFO] | +- org.spark-project:pyrolite:jar:2.0.1:provided
[INFO] | \- net.sf.py4j:py4j:jar:0.8.2.1:provided
[INFO] +- org.apache.spark:spark-sql_2.10:jar:1.1.0:provided
[INFO] | +- (org.apache.spark:spark-core_2.10:jar:1.1.0:provided - omitted for duplicate)
[INFO] | +- org.apache.spark:spark-catalyst_2.10:jar:1.1.0:provided
[INFO] | | +- org.scala-lang:scala-compiler:jar:2.10.4:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | | \- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | +- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | +- org.scalamacros:quasiquotes_2.10:jar:2.0.1:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | | \- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | \- (org.apache.spark:spark-core_2.10:jar:1.1.0:provided - omitted for duplicate)
[INFO] | +- (com.twitter:parquet-column:jar:1.4.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- (com.twitter:parquet-hadoop:jar:1.4.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.0:provided
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:provided
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.3.0:provided
[INFO] +- io.netty:netty:jar:3.6.6.Final:compile
[INFO] +- com.google.guava:guava:jar:14.0:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:2.4.1:compile
[INFO] +- org.apache.avro:avro:jar:1.7.6-cdh5.2.0:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | | \- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile - omitted for duplicate)
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.0.5.3:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] +- org.apache.avro:avro-tools:jar:1.7.6-cdh5.2.0:compile
[INFO] | \- (org.slf4j:slf4j-api:jar:1.6.4:compile - omitted for conflict with 1.7.5)
[INFO] +- com.twitter:parquet-avro:jar:1.5.0-cdh5.2.0:compile
[INFO] | +- com.twitter:parquet-column:jar:1.4.3:compile
[INFO] | | +- com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile
[INFO] | | +- com.twitter:parquet-encoding:jar:1.5.0-cdh5.2.0:compile
[INFO] | | | +- (com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | | +- com.twitter:parquet-generator:jar:1.5.0-cdh5.2.0:compile
[INFO] | | | | \- (com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | | \- (commons-codec:commons-codec:jar:1.4:compile - omitted for conflict with 1.7)
[INFO] | | \- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- com.twitter:parquet-hadoop:jar:1.4.3:compile
[INFO] | | +- (com.twitter:parquet-column:jar:1.5.0-cdh5.2.0:compile - omitted for conflict with 1.4.3)
[INFO] | | +- (com.twitter:parquet-format:jar:2.1.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | +- com.twitter:parquet-jackson:jar:1.5.0-cdh5.2.0:compile
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.11:compile - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.11:compile - omitted for conflict with 1.9.13)
[INFO] | | \- (org.xerial.snappy:snappy-java:jar:1.0.5:compile - omitted for conflict with 1.0.5.3)
[INFO] | +- com.twitter:parquet-format:jar:2.1.0-cdh5.2.0:compile
[INFO] | \- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:compile - omitted for duplicate)
[INFO] +- org.scalatest:scalatest_2.10:jar:2.2.1:test
[INFO] | +- (org.scala-lang:scala-library:jar:2.10.4:test - omitted for duplicate)
[INFO] | \- org.scala-lang:scala-reflect:jar:2.10.4:test
[INFO] | \- (org.scala-lang:scala-library:jar:2.10.4:test - omitted for duplicate)
[INFO] +- org.apache.avro:avro-mapred:jar:1.7.6-cdh5.2.0:provided
[INFO] | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | +- org.apache.avro:avro-ipc:jar:1.7.6-cdh5.2.0:provided
[INFO] | | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- org.apache.velocity:velocity:jar:1.7:provided
[INFO] | | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | | \- (commons-lang:commons-lang:jar:2.4:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- org.apache.avro:avro-ipc:jar:tests:1.7.6-cdh5.2.0:provided
[INFO] | | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.apache.velocity:velocity:jar:1.7:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] +- org.apache.hbase:hbase-server:jar:0.98.1-hadoop1:provided
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (commons-codec:commons-codec:jar:1.7:provided - omitted for conflict with 1.7)
[INFO] | | +- (commons-lang:commons-lang:jar:2.6:provided - omitted for duplicate)
[INFO] | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | +- (commons-io:commons-io:jar:2.4:provided - omitted for conflict with 2.1)
[INFO] | | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-protocol:jar:0.98.1-hadoop1:provided
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- (org.apache.hbase:hbase-client:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:0.98.1-hadoop1:provided
[INFO] | | +- (org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hbase:hbase-hadoop1-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided
[INFO] | | +- (commons-logging:commons-logging:jar:1.0.4:provided - omitted for conflict with 1.1.1)
[INFO] | | \- (commons-codec:commons-codec:jar:1.2:provided - omitted for conflict with 1.7)
[INFO] | +- commons-collections:commons-collections:jar:3.2.1:provided
[INFO] | +- org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.1-hadoop1:provided
[INFO] | | +- (org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (com.yammer.metrics:metrics-core:jar:2.1.2:provided - omitted for duplicate)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.1.2:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | +- commons-cli:commons-cli:jar:1.2:provided
[INFO] | +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.1:provided
[INFO] | +- commons-io:commons-io:jar:2.4:provided
[INFO] | +- commons-lang:commons-lang:jar:2.6:provided
[INFO] | +- commons-logging:commons-logging:jar:1.1.1:provided
[INFO] | +- org.apache.commons:commons-math:jar:2.1:provided
[INFO] | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.6.1:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (org.slf4j:slf4j-log4j12:jar:1.6.1:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (log4j:log4j:jar:1.2.16:provided - omitted for conflict with 1.2.17)
[INFO] | | \- (io.netty:netty:jar:3.7.0.Final:provided - omitted for conflict with 3.6.6.Final)
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided
[INFO] | | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided
[INFO] | | \- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:provided
[INFO] | | \- (org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:provided
[INFO] | | \- (org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:provided
[INFO] | +- (org.codehaus.jackson:jackson-core-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:provided
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- tomcat:jasper-compiler:jar:5.5.23:provided
[INFO] | +- tomcat:jasper-runtime:jar:5.5.23:provided
[INFO] | | \- (commons-el:commons-el:jar:1.0:provided - omitted for duplicate)
[INFO] | +- org.jamon:jamon-runtime:jar:2.3.1:provided
[INFO] | +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] | +- com.sun.jersey:jersey-core:jar:1.8:provided
[INFO] | +- com.sun.jersey:jersey-json:jar:1.8:provided
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
[INFO] | | | \- (javax.xml.bind:jaxb-api:jar:2.2.2:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:provided - omitted for conflict with 1.8.8)
[INFO] | | +- org.codehaus.jackson:jackson-xc:jar:1.7.1:provided
[INFO] | | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | \- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- com.sun.jersey:jersey-server:jar:1.8:provided
[INFO] | | +- asm:asm:jar:3.1:provided
[INFO] | | \- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] | | \- javax.activation:activation:jar:1.1:provided
[INFO] | +- org.cloudera.htrace:htrace-core:jar:2.04:provided
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided
[INFO] | \- junit:junit:jar:4.11:provided
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:provided
[INFO] \- org.apache.hbase:hbase-client:jar:0.98.1-hadoop1:provided
[INFO] +- (org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] +- (org.apache.hbase:hbase-protocol:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] +- (commons-codec:commons-codec:jar:1.7:compile - scope updated from provided; omitted for duplicate)
[INFO] +- (commons-io:commons-io:jar:2.4:provided - omitted for duplicate)
[INFO] +- (commons-lang:commons-lang:jar:2.6:provided - omitted for duplicate)
[INFO] +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] +- (org.apache.zookeeper:zookeeper:jar:3.4.6:provided - omitted for duplicate)
[INFO] +- (org.cloudera.htrace:htrace-core:jar:2.04:provided - omitted for duplicate)
[INFO] +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] \- (junit:junit:jar:4.11:provided - omitted for duplicate)
> From: sowen@cloudera.com
> Date: Sat, 24 Jan 2015 09:46:02 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> Hadoop 2's artifact is hadoop-common rather than hadoop-core but I
> assume you looked for that too. To answer your earlier question, no,
> Spark works with both Hadoop 1 and Hadoop 2 and is source-compatible
> with both. It can't be binary-compatible with both at once though. The
> code you cite is correct; there is no bug there.
>
> Your first error definitely indicates you have the wrong version of
> Hadoop on the client side. It's not matching your HDFS version. And
> the second suggests you are mixing code compiled for different
> versions of Hadoop. I think you need to check what version of Hadoop
> your Spark is compiled for. For example I saw a reference to CDH 5.2
> which is Hadoop 2.5, but then you're showing that you are running an
> old Hadoop 1.x HDFS? there seem to be a number of possible
> incompatibilities here.
>
> On Fri, Jan 23, 2015 at 11:38 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Sorry I still did not quiet get your resolution. In my jar, there are
> > following three related classes:
> >
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl$DummyReporter.class
> > org/apache/hadoop/mapreduce/TaskAttemptContext.class
> >
> > I think the first two come from hadoop2 and the third from hadoop1. I would
> > like to get rid of the first two. I checked my source code. It does have a
> > place using the class (or interface in hadoop2) TaskAttemptContext.
> > Do you mean I make a separate jar for this portion of code and built with
> > hadoop1 to get rid of dependency? An alternative way is to modify the code
> > in SparkHadoopMapReduceUtil.scala and put it into my own source code to
> > bypass the problem. Any comment on this? Thanks.
> >
> > ________________________________
> > From: eychih@hotmail.com
> > To: sowen@cloudera.com
> > CC: user@spark.apache.org
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > Date: Fri, 23 Jan 2015 11:17:36 -0800
> >
> >
> > Thanks. I looked at the dependency tree. I did not see any dependent jar
> > of hadoop-core from hadoop2. However the jar built from maven has the
> > class:
> >
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
> >
> > Do you know why?
> >
> >
> >
> >
> > ________________________________
> > Date: Fri, 23 Jan 2015 17:01:48 +0000
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > From: sowen@cloudera.com
> > To: eychih@hotmail.com
> >
> > Are you receiving my replies? I have suggested a resolution. Look at the
> > dependency tree next.
> >
> > On Jan 23, 2015 2:43 PM, "ey-chih chow" <ey...@hotmail.com> wrote:
> >
> > I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it
> > is broken in the following code:
> >
> > def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID):
> > TaskAttemptContext = {
> > val klass = firstAvailableClass(
> > "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", //
> > hadoop2, hadoop2-yarn
> > "org.apache.hadoop.mapreduce.TaskAttemptContext") //
> > hadoop1
> > val ctor = klass.getDeclaredConstructor(classOf[Configuration],
> > classOf[TaskAttemptID])
> > ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext]
> > }
> >
> > In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any
> > suggestion how to resolve it?
> >
> > Thanks.
> >
> >
> >
> >> From: sowen@cloudera.com
> >> Date: Fri, 23 Jan 2015 14:01:45 +0000
> >> Subject: Re: spark 1.1.0 save data to hdfs failed
> >> To: eychih@hotmail.com
> >> CC: user@spark.apache.org
> >>
> >> These are all definitely symptoms of mixing incompatible versions of
> >> libraries.
> >>
> >> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> >> not the only way Hadoop deps get into your app. See my suggestion
> >> about investigating the dependency tree.
> >>
> >> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> >> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> >> > provided. Why the cluster's version is not used?
> >> >
> >> > Any way, as I mentioned in the previous message, after changing the
> >> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> >> > exception and go to another one as indicated below. Any suggestion on
> >> > this?
> >> >
> >> > =================================
> >> >
> >> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> > at
> >> >
> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> > at java.lang.reflect.Method.invoke(Method.java:606)
> >> > at
> >> >
> >> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> >> > at
> >> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> >> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> >> > at
> >> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> >> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> >> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:191)
> >> > at
> >> >
> >> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> >> > at
> >> >
> >> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> >> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> >> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >> >
> >> > ... 6 more
> >> >
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
Thanks for the information. I changed the dependencies for Spark jars as follows:
<dependency> <!-- Spark -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- Spark SQL -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.1.0</version>
<scope>provided</scope>
</dependency>
I don't know how these libraries are built, but I saw Spark has maven pom files. I think these jars should be built from the corresponding pom files. These pom files have dependencies on hadoop version 1.0.4. So I don't know where the hadoop2 jar come from. What follows is a major fragment of my current dependency tree. I don't know where the hadoop2 classes come into my built jar.
======================
[INFO] | \- org.apache.hadoop:hadoop-core:jar:1.2.1:provided
[INFO] | +- xmlenc:xmlenc:jar:0.52:provided
[INFO] | +- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (com.sun.jersey:jersey-json:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (com.sun.jersey:jersey-server:jar:1.8:provided - omitted for duplicate)
[INFO] | +- (commons-io:commons-io:jar:2.1:provided - omitted for conflict with 2.4)
[INFO] | +- (commons-codec:commons-codec:jar:1.4:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- (org.apache.commons:commons-math:jar:2.1:provided - omitted for duplicate)
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided
[INFO] | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | +- (commons-lang:commons-lang:jar:2.4:provided - omitted for conflict with 2.6)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- commons-digester:commons-digester:jar:1.8:provided
[INFO] | | | +- commons-beanutils:commons-beanutils:jar:1.7.0:provided
[INFO] | | | | \- (commons-logging:commons-logging:jar:1.0.3:provided - omitted for conflict with 1.1.1)
[INFO] | | | \- (commons-logging:commons-logging:jar:1.1:provided - omitted for conflict with 1.1.1)
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:provided
[INFO] | | \- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | +- (commons-net:commons-net:jar:1.4.1:provided - omitted for conflict with 2.2)
[INFO] | +- commons-el:commons-el:jar:1.0:provided
[INFO] | | \- (commons-logging:commons-logging:jar:1.0.3:provided - omitted for conflict with 1.1.1)
[INFO] | +- hsqldb:hsqldb:jar:1.8.0.10:provided
[INFO] | +- oro:oro:jar:2.0.8:provided
[INFO] | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.1.0:provided
[INFO] | +- (org.apache.hadoop:hadoop-client:jar:1.0.4:provided - omitted for conflict with 1.2.1)
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:provided
[INFO] | | +- (commons-codec:commons-codec:jar:1.3:provided - omitted for conflict with 1.4)
[INFO] | | \- (commons-httpclient:commons-httpclient:jar:3.1:provided - omitted for duplicate)
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:provided
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:provided
[INFO] | | | +- org.apache.curator:curator-client:jar:2.4.0:provided
[INFO] | | | | +- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.6.1)
[INFO] | | | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for duplicate)
[INFO] | | | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for duplicate)
[INFO] | | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.zookeeper:zookeeper:jar:3.4.5:provided - omitted for conflict with 3.4.6)
[INFO] | | \- (com.google.guava:guava:jar:14.0.1:provided - omitted for duplicate)
[INFO] | +- org.eclipse.jetty:jetty-plus:jar:8.1.14.v20131031:provided
[INFO] | | +- org.eclipse.jetty.orbit:javax.transaction:jar:1.1.1.v201105210645:provided
[INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:8.1.14.v20131031:provided
[INFO] | | | +- org.eclipse.jetty:jetty-xml:jar:8.1.14.v20131031:provided
[INFO] | | | | \- (org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:provided
[INFO] | | | \- (org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | \- org.eclipse.jetty:jetty-jndi:jar:8.1.14.v20131031:provided
[INFO] | | +- (org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | | \- org.eclipse.jetty.orbit:javax.mail.glassfish:jar:1.4.1.v201005082020:provided
[INFO] | | \- org.eclipse.jetty.orbit:javax.activation:jar:1.1.0.v201105071233:provided
[INFO] | +- org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:provided
[INFO] | | \- (org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | +- org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided
[INFO] | +- org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:provided
[INFO] | | +- org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:provided
[INFO] | | +- org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:provided
[INFO] | | \- org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:provided
[INFO] | | \- org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:provided
[INFO] | | \- (org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:provided - omitted for duplicate)
[INFO] | +- (com.google.guava:guava:jar:14.0.1:provided - omitted for conflict with 14.0)
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:provided
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:provided
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.5:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.5:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.5:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- log4j:log4j:jar:1.2.17:provided
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | | \- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | +- com.ning:compress-lzf:jar:1.0.0:provided
[INFO] | +- (org.xerial.snappy:snappy-java:jar:1.0.5.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- net.jpountz.lz4:lz4:jar:1.2.0:provided
[INFO] | +- com.twitter:chill_2.10:jar:0.3.6:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.3:provided - omitted for conflict with 2.10.4)
[INFO] | | +- (com.twitter:chill-java:jar:0.3.6:provided - omitted for duplicate)
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:provided
[INFO] | | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:provided
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:provided
[INFO] | | \- org.objenesis:objenesis:jar:1.2:provided
[INFO] | +- com.twitter:chill-java:jar:0.3.6:provided
[INFO] | | \- (com.esotericsoftware.kryo:kryo:jar:2.21:provided - omitted for duplicate)
[INFO] | +- commons-net:commons-net:jar:2.2:provided
[INFO] | +- org.spark-project.akka:akka-remote_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | +- org.spark-project.akka:akka-actor_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | | \- com.typesafe:config:jar:1.0.2:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | +- (io.netty:netty:jar:3.6.6.Final:provided - omitted for duplicate)
[INFO] | | +- org.spark-project.protobuf:protobuf-java:jar:2.4.1-shaded:provided
[INFO] | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:provided
[INFO] | +- org.spark-project.akka:akka-slf4j_2.10:jar:2.2.3-shaded-protobuf:provided
[INFO] | | +- (org.spark-project.akka:akka-actor_2.10:jar:2.2.3-shaded-protobuf:provided - omitted for duplicate)
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.2:provided - omitted for conflict with 2.10.3)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | +- org.scala-lang:scala-library:jar:2.10.4:provided
[INFO] | +- org.json4s:json4s-jackson_2.10:jar:3.2.10:provided
[INFO] | | +- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | +- org.json4s:json4s-core_2.10:jar:3.2.10:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:provided
[INFO] | | | | \- (org.scala-lang:scala-library:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | | +- (com.thoughtworks.paranamer:paranamer:jar:2.6:provided - omitted for conflict with 2.3)
[INFO] | | | \- org.scala-lang:scalap:jar:2.10.0:provided
[INFO] | | | \- (org.scala-lang:scala-compiler:jar:2.10.0:provided - omitted for conflict with 2.10.4)
[INFO] | | \- (com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:provided - omitted for conflict with 2.3.0)
[INFO] | +- colt:colt:jar:1.2.0:provided
[INFO] | | \- concurrent:concurrent:jar:1.3.4:provided
[INFO] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.18.1:provided
[INFO] | +- io.netty:netty-all:jar:4.0.23.Final:provided
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] | +- com.codahale.metrics:metrics-core:jar:3.0.0:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-jvm:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-json:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.2:provided - omitted for conflict with 2.3.1)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-graphite:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-jvm:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-json:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | +- (com.fasterxml.jackson.core:jackson-databind:jar:2.2.2:provided - omitted for conflict with 2.3.1)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- com.codahale.metrics:metrics-graphite:jar:3.0.0:provided
[INFO] | | +- (com.codahale.metrics:metrics-core:jar:3.0.0:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.7.5:provided - omitted for duplicate)
[INFO] | +- org.tachyonproject:tachyon-client:jar:0.5.0:provided
[INFO] | | \- org.tachyonproject:tachyon:jar:0.5.0:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (org.slf4j:slf4j-log4j12:jar:1.7.2:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | +- (commons-io:commons-io:jar:2.4:provided - omitted for conflict with 2.1)
[INFO] | | +- (org.apache.commons:commons-lang3:jar:3.0:provided - omitted for conflict with 3.3.2)
[INFO] | | \- (com.fasterxml.jackson.core:jackson-databind:jar:2.3.0:provided - omitted for conflict with 2.3.1)
[INFO] | +- org.spark-project:pyrolite:jar:2.0.1:provided
[INFO] | \- net.sf.py4j:py4j:jar:0.8.2.1:provided
[INFO] +- org.apache.spark:spark-sql_2.10:jar:1.1.0:provided
[INFO] | +- (org.apache.spark:spark-core_2.10:jar:1.1.0:provided - omitted for duplicate)
[INFO] | +- org.apache.spark:spark-catalyst_2.10:jar:1.1.0:provided
[INFO] | | +- org.scala-lang:scala-compiler:jar:2.10.4:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | | \- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | +- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | +- org.scalamacros:quasiquotes_2.10:jar:2.0.1:provided
[INFO] | | | +- (org.scala-lang:scala-library:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | | \- (org.scala-lang:scala-reflect:jar:2.10.4:provided - omitted for duplicate)
[INFO] | | \- (org.apache.spark:spark-core_2.10:jar:1.1.0:provided - omitted for duplicate)
[INFO] | +- (com.twitter:parquet-column:jar:1.4.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | +- (com.twitter:parquet-hadoop:jar:1.4.3:compile - scope updated from provided; omitted for duplicate)
[INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.0:provided
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:provided
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.3.0:provided
[INFO] +- io.netty:netty:jar:3.6.6.Final:compile
[INFO] +- com.google.guava:guava:jar:14.0:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:2.4.1:compile
[INFO] +- org.apache.avro:avro:jar:1.7.6-cdh5.2.0:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] | | \- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile - omitted for duplicate)
[INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.0.5.3:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | \- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] +- org.apache.avro:avro-tools:jar:1.7.6-cdh5.2.0:compile
[INFO] | \- (org.slf4j:slf4j-api:jar:1.6.4:compile - omitted for conflict with 1.7.5)
[INFO] +- com.twitter:parquet-avro:jar:1.5.0-cdh5.2.0:compile
[INFO] | +- com.twitter:parquet-column:jar:1.4.3:compile
[INFO] | | +- com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile
[INFO] | | +- com.twitter:parquet-encoding:jar:1.5.0-cdh5.2.0:compile
[INFO] | | | +- (com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | | +- com.twitter:parquet-generator:jar:1.5.0-cdh5.2.0:compile
[INFO] | | | | \- (com.twitter:parquet-common:jar:1.5.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | | \- (commons-codec:commons-codec:jar:1.4:compile - omitted for conflict with 1.7)
[INFO] | | \- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- com.twitter:parquet-hadoop:jar:1.4.3:compile
[INFO] | | +- (com.twitter:parquet-column:jar:1.5.0-cdh5.2.0:compile - omitted for conflict with 1.4.3)
[INFO] | | +- (com.twitter:parquet-format:jar:2.1.0-cdh5.2.0:compile - omitted for duplicate)
[INFO] | | +- com.twitter:parquet-jackson:jar:1.5.0-cdh5.2.0:compile
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.11:compile - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.11:compile - omitted for conflict with 1.9.13)
[INFO] | | \- (org.xerial.snappy:snappy-java:jar:1.0.5:compile - omitted for conflict with 1.0.5.3)
[INFO] | +- com.twitter:parquet-format:jar:2.1.0-cdh5.2.0:compile
[INFO] | \- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:compile - omitted for duplicate)
[INFO] +- org.scalatest:scalatest_2.10:jar:2.2.1:test
[INFO] | +- (org.scala-lang:scala-library:jar:2.10.4:test - omitted for duplicate)
[INFO] | \- org.scala-lang:scala-reflect:jar:2.10.4:test
[INFO] | \- (org.scala-lang:scala-library:jar:2.10.4:test - omitted for duplicate)
[INFO] +- org.apache.avro:avro-mapred:jar:1.7.6-cdh5.2.0:provided
[INFO] | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | +- org.apache.avro:avro-ipc:jar:1.7.6-cdh5.2.0:provided
[INFO] | | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- org.apache.velocity:velocity:jar:1.7:provided
[INFO] | | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | | \- (commons-lang:commons-lang:jar:2.4:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- org.apache.avro:avro-ipc:jar:tests:1.7.6-cdh5.2.0:provided
[INFO] | | +- (org.apache.avro:avro:jar:1.7.6-cdh5.2.0:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | | +- (org.apache.velocity:velocity:jar:1.7:provided - omitted for duplicate)
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- (org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided - omitted for duplicate)
[INFO] | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] +- org.apache.hbase:hbase-server:jar:0.98.1-hadoop1:provided
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (commons-codec:commons-codec:jar:1.7:provided - omitted for conflict with 1.7)
[INFO] | | +- (commons-lang:commons-lang:jar:2.6:provided - omitted for duplicate)
[INFO] | | +- (commons-collections:commons-collections:jar:3.2.1:provided - omitted for duplicate)
[INFO] | | +- (commons-io:commons-io:jar:2.4:provided - omitted for conflict with 2.1)
[INFO] | | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-protocol:jar:0.98.1-hadoop1:provided
[INFO] | | +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- (org.apache.hbase:hbase-client:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:0.98.1-hadoop1:provided
[INFO] | | +- (org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hbase:hbase-hadoop1-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:provided
[INFO] | | +- (commons-logging:commons-logging:jar:1.0.4:provided - omitted for conflict with 1.1.1)
[INFO] | | \- (commons-codec:commons-codec:jar:1.2:provided - omitted for conflict with 1.7)
[INFO] | +- commons-collections:commons-collections:jar:3.2.1:provided
[INFO] | +- org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-hadoop1-compat:jar:0.98.1-hadoop1:provided
[INFO] | | +- (org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] | | +- (com.yammer.metrics:metrics-core:jar:2.1.2:provided - omitted for duplicate)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] | | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | | \- (junit:junit:jar:4.11:provided - omitted for duplicate)
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.1.2:provided
[INFO] | | \- (org.slf4j:slf4j-api:jar:1.6.4:provided - omitted for conflict with 1.7.5)
[INFO] | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | +- commons-cli:commons-cli:jar:1.2:provided
[INFO] | +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.1:provided
[INFO] | +- commons-io:commons-io:jar:2.4:provided
[INFO] | +- commons-lang:commons-lang:jar:2.6:provided
[INFO] | +- commons-logging:commons-logging:jar:1.1.1:provided
[INFO] | +- org.apache.commons:commons-math:jar:2.1:provided
[INFO] | +- (log4j:log4j:jar:1.2.17:provided - omitted for duplicate)
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.6.1:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (org.slf4j:slf4j-log4j12:jar:1.6.1:provided - omitted for conflict with 1.7.5)
[INFO] | | +- (log4j:log4j:jar:1.2.16:provided - omitted for conflict with 1.2.17)
[INFO] | | \- (io.netty:netty:jar:3.7.0.Final:provided - omitted for conflict with 3.6.6.Final)
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided
[INFO] | | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided
[INFO] | | \- (org.mortbay.jetty:jetty:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:provided
[INFO] | | \- (org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:provided
[INFO] | | \- (org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:provided - omitted for duplicate)
[INFO] | +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:provided
[INFO] | +- (org.codehaus.jackson:jackson-core-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:provided
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] | +- tomcat:jasper-compiler:jar:5.5.23:provided
[INFO] | +- tomcat:jasper-runtime:jar:5.5.23:provided
[INFO] | | \- (commons-el:commons-el:jar:1.0:provided - omitted for duplicate)
[INFO] | +- org.jamon:jamon-runtime:jar:2.3.1:provided
[INFO] | +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] | +- com.sun.jersey:jersey-core:jar:1.8:provided
[INFO] | +- com.sun.jersey:jersey-json:jar:1.8:provided
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided
[INFO] | | | \- (javax.xml.bind:jaxb-api:jar:2.2.2:provided - omitted for duplicate)
[INFO] | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | +- (org.codehaus.jackson:jackson-jaxrs:jar:1.7.1:provided - omitted for conflict with 1.8.8)
[INFO] | | +- org.codehaus.jackson:jackson-xc:jar:1.7.1:provided
[INFO] | | | +- (org.codehaus.jackson:jackson-core-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | | \- (org.codehaus.jackson:jackson-mapper-asl:jar:1.7.1:provided - omitted for conflict with 1.9.13)
[INFO] | | \- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- com.sun.jersey:jersey-server:jar:1.8:provided
[INFO] | | +- asm:asm:jar:3.1:provided
[INFO] | | \- (com.sun.jersey:jersey-core:jar:1.8:provided - omitted for duplicate)
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] | | \- javax.activation:activation:jar:1.1:provided
[INFO] | +- org.cloudera.htrace:htrace-core:jar:2.04:provided
[INFO] | | +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] | | +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] | | \- (org.mortbay.jetty:jetty-util:jar:6.1.26:provided - omitted for duplicate)
[INFO] | +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] | +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided
[INFO] | \- junit:junit:jar:4.11:provided
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:provided
[INFO] \- org.apache.hbase:hbase-client:jar:0.98.1-hadoop1:provided
[INFO] +- (org.apache.hbase:hbase-common:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] +- (org.apache.hbase:hbase-protocol:jar:0.98.1-hadoop1:provided - omitted for duplicate)
[INFO] +- (commons-codec:commons-codec:jar:1.7:compile - scope updated from provided; omitted for duplicate)
[INFO] +- (commons-io:commons-io:jar:2.4:provided - omitted for duplicate)
[INFO] +- (commons-lang:commons-lang:jar:2.6:provided - omitted for duplicate)
[INFO] +- (commons-logging:commons-logging:jar:1.1.1:provided - omitted for duplicate)
[INFO] +- (com.google.guava:guava:jar:12.0.1:provided - omitted for conflict with 14.0)
[INFO] +- (com.google.protobuf:protobuf-java:jar:2.5.0:provided - omitted for conflict with 2.4.1)
[INFO] +- (org.apache.zookeeper:zookeeper:jar:3.4.6:provided - omitted for duplicate)
[INFO] +- (org.cloudera.htrace:htrace-core:jar:2.04:provided - omitted for duplicate)
[INFO] +- (org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:provided - omitted for conflict with 1.9.13)
[INFO] +- (org.apache.hadoop:hadoop-core:jar:1.2.1:provided - omitted for duplicate)
[INFO] +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:provided - omitted for duplicate)
[INFO] \- (junit:junit:jar:4.11:provided - omitted for duplicate)
> From: sowen@cloudera.com
> Date: Sat, 24 Jan 2015 09:46:02 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> Hadoop 2's artifact is hadoop-common rather than hadoop-core but I
> assume you looked for that too. To answer your earlier question, no,
> Spark works with both Hadoop 1 and Hadoop 2 and is source-compatible
> with both. It can't be binary-compatible with both at once though. The
> code you cite is correct; there is no bug there.
>
> Your first error definitely indicates you have the wrong version of
> Hadoop on the client side. It's not matching your HDFS version. And
> the second suggests you are mixing code compiled for different
> versions of Hadoop. I think you need to check what version of Hadoop
> your Spark is compiled for. For example I saw a reference to CDH 5.2
> which is Hadoop 2.5, but then you're showing that you are running an
> old Hadoop 1.x HDFS? there seem to be a number of possible
> incompatibilities here.
>
> On Fri, Jan 23, 2015 at 11:38 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Sorry I still did not quiet get your resolution. In my jar, there are
> > following three related classes:
> >
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl$DummyReporter.class
> > org/apache/hadoop/mapreduce/TaskAttemptContext.class
> >
> > I think the first two come from hadoop2 and the third from hadoop1. I would
> > like to get rid of the first two. I checked my source code. It does have a
> > place using the class (or interface in hadoop2) TaskAttemptContext.
> > Do you mean I make a separate jar for this portion of code and built with
> > hadoop1 to get rid of dependency? An alternative way is to modify the code
> > in SparkHadoopMapReduceUtil.scala and put it into my own source code to
> > bypass the problem. Any comment on this? Thanks.
> >
> > ________________________________
> > From: eychih@hotmail.com
> > To: sowen@cloudera.com
> > CC: user@spark.apache.org
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > Date: Fri, 23 Jan 2015 11:17:36 -0800
> >
> >
> > Thanks. I looked at the dependency tree. I did not see any dependent jar
> > of hadoop-core from hadoop2. However the jar built from maven has the
> > class:
> >
> > org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
> >
> > Do you know why?
> >
> >
> >
> >
> > ________________________________
> > Date: Fri, 23 Jan 2015 17:01:48 +0000
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > From: sowen@cloudera.com
> > To: eychih@hotmail.com
> >
> > Are you receiving my replies? I have suggested a resolution. Look at the
> > dependency tree next.
> >
> > On Jan 23, 2015 2:43 PM, "ey-chih chow" <ey...@hotmail.com> wrote:
> >
> > I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it
> > is broken in the following code:
> >
> > def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID):
> > TaskAttemptContext = {
> > val klass = firstAvailableClass(
> > "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", //
> > hadoop2, hadoop2-yarn
> > "org.apache.hadoop.mapreduce.TaskAttemptContext") //
> > hadoop1
> > val ctor = klass.getDeclaredConstructor(classOf[Configuration],
> > classOf[TaskAttemptID])
> > ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext]
> > }
> >
> > In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any
> > suggestion how to resolve it?
> >
> > Thanks.
> >
> >
> >
> >> From: sowen@cloudera.com
> >> Date: Fri, 23 Jan 2015 14:01:45 +0000
> >> Subject: Re: spark 1.1.0 save data to hdfs failed
> >> To: eychih@hotmail.com
> >> CC: user@spark.apache.org
> >>
> >> These are all definitely symptoms of mixing incompatible versions of
> >> libraries.
> >>
> >> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> >> not the only way Hadoop deps get into your app. See my suggestion
> >> about investigating the dependency tree.
> >>
> >> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> >> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> >> > provided. Why the cluster's version is not used?
> >> >
> >> > Any way, as I mentioned in the previous message, after changing the
> >> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> >> > exception and go to another one as indicated below. Any suggestion on
> >> > this?
> >> >
> >> > =================================
> >> >
> >> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> > at
> >> >
> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> > at java.lang.reflect.Method.invoke(Method.java:606)
> >> > at
> >> >
> >> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> >> > at
> >> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> >> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> >> > at java.lang.ClassLoader.defineClass1(Native Method)
> >> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> >> > at
> >> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> >> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> >> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> >> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> >> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> >> > at java.lang.Class.forName0(Native Method)
> >> > at java.lang.Class.forName(Class.java:191)
> >> > at
> >> >
> >> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> >> > at
> >> >
> >> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> >> > at
> >> >
> >> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> >> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> >> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >> >
> >> > ... 6 more
> >> >
Re: spark 1.1.0 save data to hdfs failed
Posted by Sean Owen <so...@cloudera.com>.
Hadoop 2's artifact is hadoop-common rather than hadoop-core but I
assume you looked for that too. To answer your earlier question, no,
Spark works with both Hadoop 1 and Hadoop 2 and is source-compatible
with both. It can't be binary-compatible with both at once though. The
code you cite is correct; there is no bug there.
Your first error definitely indicates you have the wrong version of
Hadoop on the client side. It's not matching your HDFS version. And
the second suggests you are mixing code compiled for different
versions of Hadoop. I think you need to check what version of Hadoop
your Spark is compiled for. For example I saw a reference to CDH 5.2
which is Hadoop 2.5, but then you're showing that you are running an
old Hadoop 1.x HDFS? there seem to be a number of possible
incompatibilities here.
On Fri, Jan 23, 2015 at 11:38 PM, ey-chih chow <ey...@hotmail.com> wrote:
> Sorry I still did not quiet get your resolution. In my jar, there are
> following three related classes:
>
> org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
> org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl$DummyReporter.class
> org/apache/hadoop/mapreduce/TaskAttemptContext.class
>
> I think the first two come from hadoop2 and the third from hadoop1. I would
> like to get rid of the first two. I checked my source code. It does have a
> place using the class (or interface in hadoop2) TaskAttemptContext.
> Do you mean I make a separate jar for this portion of code and built with
> hadoop1 to get rid of dependency? An alternative way is to modify the code
> in SparkHadoopMapReduceUtil.scala and put it into my own source code to
> bypass the problem. Any comment on this? Thanks.
>
> ________________________________
> From: eychih@hotmail.com
> To: sowen@cloudera.com
> CC: user@spark.apache.org
> Subject: RE: spark 1.1.0 save data to hdfs failed
> Date: Fri, 23 Jan 2015 11:17:36 -0800
>
>
> Thanks. I looked at the dependency tree. I did not see any dependent jar
> of hadoop-core from hadoop2. However the jar built from maven has the
> class:
>
> org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
>
> Do you know why?
>
>
>
>
> ________________________________
> Date: Fri, 23 Jan 2015 17:01:48 +0000
> Subject: RE: spark 1.1.0 save data to hdfs failed
> From: sowen@cloudera.com
> To: eychih@hotmail.com
>
> Are you receiving my replies? I have suggested a resolution. Look at the
> dependency tree next.
>
> On Jan 23, 2015 2:43 PM, "ey-chih chow" <ey...@hotmail.com> wrote:
>
> I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it
> is broken in the following code:
>
> def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID):
> TaskAttemptContext = {
> val klass = firstAvailableClass(
> "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", //
> hadoop2, hadoop2-yarn
> "org.apache.hadoop.mapreduce.TaskAttemptContext") //
> hadoop1
> val ctor = klass.getDeclaredConstructor(classOf[Configuration],
> classOf[TaskAttemptID])
> ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext]
> }
>
> In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any
> suggestion how to resolve it?
>
> Thanks.
>
>
>
>> From: sowen@cloudera.com
>> Date: Fri, 23 Jan 2015 14:01:45 +0000
>> Subject: Re: spark 1.1.0 save data to hdfs failed
>> To: eychih@hotmail.com
>> CC: user@spark.apache.org
>>
>> These are all definitely symptoms of mixing incompatible versions of
>> libraries.
>>
>> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
>> not the only way Hadoop deps get into your app. See my suggestion
>> about investigating the dependency tree.
>>
>> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
>> > Thanks. But I think I already mark all the Spark and Hadoop reps as
>> > provided. Why the cluster's version is not used?
>> >
>> > Any way, as I mentioned in the previous message, after changing the
>> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
>> > exception and go to another one as indicated below. Any suggestion on
>> > this?
>> >
>> > =================================
>> >
>> > Exception in thread "main" java.lang.reflect.InvocationTargetException
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> > at
>> >
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
>> > at
>> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
>> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
>> > at java.lang.ClassLoader.defineClass1(Native Method)
>> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> > at
>> > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> > at java.lang.Class.forName0(Native Method)
>> > at java.lang.Class.forName(Class.java:191)
>> > at
>> >
>> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
>> > at
>> >
>> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
>> > at
>> >
>> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
>> > at
>> >
>> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
>> > at
>> >
>> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
>> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
>> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
>> >
>> > ... 6 more
>> >
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
Sorry I still did not quiet get your resolution. In my jar, there are following three related classes:
org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.classorg/apache/hadoop/mapreduce/task/TaskAttemptContextImpl$DummyReporter.classorg/apache/hadoop/mapreduce/TaskAttemptContext.class
I think the first two come from hadoop2 and the third from hadoop1. I would like to get rid of the first two. I checked my source code. It does have a place using the class (or interface in hadoop2) TaskAttemptContext.Do you mean I make a separate jar for this portion of code and built with hadoop1 to get rid of dependency? An alternative way is to modify the code in SparkHadoopMapReduceUtil.scala and put it into my own source code to bypass the problem. Any comment on this? Thanks.
From: eychih@hotmail.com
To: sowen@cloudera.com
CC: user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Fri, 23 Jan 2015 11:17:36 -0800
Thanks. I looked at the dependency tree. I did not see any dependent jar of hadoop-core from hadoop2. However the jar built from maven has the class:
org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
Do you know why?
Date: Fri, 23 Jan 2015 17:01:48 +0000
Subject: RE: spark 1.1.0 save data to hdfs failed
From: sowen@cloudera.com
To: eychih@hotmail.com
Are you receiving my replies? I have suggested a resolution. Look at the dependency tree next.
On Jan 23, 2015 2:43 PM, "ey-chih chow" <ey...@hotmail.com> wrote:
I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it is broken in the following code:
def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID): TaskAttemptContext = { val klass = firstAvailableClass( "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", // hadoop2, hadoop2-yarn "org.apache.hadoop.mapreduce.TaskAttemptContext") // hadoop1 val ctor = klass.getDeclaredConstructor(classOf[Configuration], classOf[TaskAttemptID]) ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext] }
In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any suggestion how to resolve it?
Thanks.
> From: sowen@cloudera.com
> Date: Fri, 23 Jan 2015 14:01:45 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> These are all definitely symptoms of mixing incompatible versions of libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the dependency tree.
>
> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> > provided. Why the cluster's version is not used?
> >
> > Any way, as I mentioned in the previous message, after changing the
> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> > exception and go to another one as indicated below. Any suggestion on this?
> >
> > =================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:191)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >
> > ... 6 more
> >
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
Thanks. I looked at the dependency tree. I did not see any dependent jar of hadoop-core from hadoop2. However the jar built from maven has the class:
org/apache/hadoop/mapreduce/task/TaskAttemptContextImpl.class
Do you know why?
Date: Fri, 23 Jan 2015 17:01:48 +0000
Subject: RE: spark 1.1.0 save data to hdfs failed
From: sowen@cloudera.com
To: eychih@hotmail.com
Are you receiving my replies? I have suggested a resolution. Look at the dependency tree next.
On Jan 23, 2015 2:43 PM, "ey-chih chow" <ey...@hotmail.com> wrote:
I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it is broken in the following code:
def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID): TaskAttemptContext = { val klass = firstAvailableClass( "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", // hadoop2, hadoop2-yarn "org.apache.hadoop.mapreduce.TaskAttemptContext") // hadoop1 val ctor = klass.getDeclaredConstructor(classOf[Configuration], classOf[TaskAttemptID]) ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext] }
In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any suggestion how to resolve it?
Thanks.
> From: sowen@cloudera.com
> Date: Fri, 23 Jan 2015 14:01:45 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> These are all definitely symptoms of mixing incompatible versions of libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the dependency tree.
>
> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> > provided. Why the cluster's version is not used?
> >
> > Any way, as I mentioned in the previous message, after changing the
> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> > exception and go to another one as indicated below. Any suggestion on this?
> >
> > =================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:191)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >
> > ... 6 more
> >
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
I looked into the source code of SparkHadoopMapReduceUtil.scala. I think it is broken in the following code:
def newTaskAttemptContext(conf: Configuration, attemptId: TaskAttemptID): TaskAttemptContext = { val klass = firstAvailableClass( "org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl", // hadoop2, hadoop2-yarn "org.apache.hadoop.mapreduce.TaskAttemptContext") // hadoop1 val ctor = klass.getDeclaredConstructor(classOf[Configuration], classOf[TaskAttemptID]) ctor.newInstance(conf, attemptId).asInstanceOf[TaskAttemptContext] }
In other words, it is related to hadoop2, hadoop2-yarn, and hadoop1. Any suggestion how to resolve it?
Thanks.
> From: sowen@cloudera.com
> Date: Fri, 23 Jan 2015 14:01:45 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> These are all definitely symptoms of mixing incompatible versions of libraries.
>
> I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
> not the only way Hadoop deps get into your app. See my suggestion
> about investigating the dependency tree.
>
> On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > Thanks. But I think I already mark all the Spark and Hadoop reps as
> > provided. Why the cluster's version is not used?
> >
> > Any way, as I mentioned in the previous message, after changing the
> > hadoop-client to version 1.2.1 in my maven deps, I already pass the
> > exception and go to another one as indicated below. Any suggestion on this?
> >
> > =================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> > at java.lang.ClassLoader.defineClass1(Native Method)
> > at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:191)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> > at
> > org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> >
> > ... 6 more
> >
Re: spark 1.1.0 save data to hdfs failed
Posted by Sean Owen <so...@cloudera.com>.
These are all definitely symptoms of mixing incompatible versions of libraries.
I'm not suggesting you haven't excluded Spark / Hadoop, but, this is
not the only way Hadoop deps get into your app. See my suggestion
about investigating the dependency tree.
On Fri, Jan 23, 2015 at 1:53 PM, ey-chih chow <ey...@hotmail.com> wrote:
> Thanks. But I think I already mark all the Spark and Hadoop reps as
> provided. Why the cluster's version is not used?
>
> Any way, as I mentioned in the previous message, after changing the
> hadoop-client to version 1.2.1 in my maven deps, I already pass the
> exception and go to another one as indicated below. Any suggestion on this?
>
> =================================
>
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:191)
> at
> org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
> at
> org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
> at
> org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
> at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
>
> ... 6 more
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
Thanks. But I think I already mark all the Spark and Hadoop reps as provided. Why the cluster's version is not used?
Any way, as I mentioned in the previous message, after changing the hadoop-client to version 1.2.1 in my maven deps, I already pass the exception and go to another one as indicated below. Any suggestion on this?
=================================Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:191)
at org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.firstAvailableClass(SparkHadoopMapReduceUtil.scala:73)
at org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil$class.newTaskAttemptContext(SparkHadoopMapReduceUtil.scala:35)
at org.apache.spark.rdd.PairRDDFunctions.newTaskAttemptContext(PairRDDFunctions.scala:53)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:932)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:103)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala) ... 6 more
> From: sowen@cloudera.com
> Date: Fri, 23 Jan 2015 10:41:12 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> So, you should not depend on Hadoop artifacts unless you use them
> directly. You should mark Hadoop and Spark deps as provided. Then the
> cluster's version is used at runtime with spark-submit. That's the
> usual way to do it, which works.
>
> If you need to embed Spark in your app and are running it outside the
> cluster for some reason, and you have to embed Hadoop and Spark code
> in your app, the version has to match. You should also use mvn
> dependency:tree to see all the dependencies coming in. There may be
> many sources of a Hadoop dep.
>
> On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow <ey...@hotmail.com> wrote:
> > Thanks. But after I replace the maven dependence from
> >
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-client</artifactId>
> > <version>2.5.0-cdh5.2.0</version>
> > <scope>provided</scope>
> > <exclusions>
> > <exclusion>
> > <groupId>org.mortbay.jetty</groupId>
> > <artifactId>servlet-api</artifactId>
> > </exclusion>
> > <exclusion>
> > <groupId>javax.servlet</groupId>
> > <artifactId>servlet-api</artifactId>
> > </exclusion>
> > <exclusion>
> > <groupId>io.netty</groupId>
> > <artifactId>netty</artifactId>
> > </exclusion>
> > </exclusions>
> > </dependency>
> >
> > to
> >
> > <dependency>
> >
> > <groupId>org.apache.hadoop</groupId>
> >
> > <artifactId>hadoop-client</artifactId>
> >
> > <version>1.0.4</version>
> >
> > <scope>provided</scope>
> >
> > <exclusions>
> >
> > <exclusion>
> >
> > <groupId>org.mortbay.jetty</groupId>
> >
> > <artifactId>servlet-api</artifactId>
> >
> > </exclusion>
> >
> > <exclusion>
> >
> > <groupId>javax.servlet</groupId>
> >
> > <artifactId>servlet-api</artifactId>
> >
> > </exclusion>
> >
> > <exclusion>
> >
> > <groupId>io.netty</groupId>
> >
> > <artifactId>netty</artifactId>
> >
> > </exclusion>
> >
> > </exclusions>
> >
> > </dependency>
> >
> >
> > the warning message is still shown up in the namenode log. Is there any
> > other thing I need to do?
> >
> >
> > Thanks.
> >
> >
> > Ey-Chih Chow
> >
> >
Re: spark 1.1.0 save data to hdfs failed
Posted by Sean Owen <so...@cloudera.com>.
So, you should not depend on Hadoop artifacts unless you use them
directly. You should mark Hadoop and Spark deps as provided. Then the
cluster's version is used at runtime with spark-submit. That's the
usual way to do it, which works.
If you need to embed Spark in your app and are running it outside the
cluster for some reason, and you have to embed Hadoop and Spark code
in your app, the version has to match. You should also use mvn
dependency:tree to see all the dependencies coming in. There may be
many sources of a Hadoop dep.
On Fri, Jan 23, 2015 at 1:05 AM, ey-chih chow <ey...@hotmail.com> wrote:
> Thanks. But after I replace the maven dependence from
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>2.5.0-cdh5.2.0</version>
> <scope>provided</scope>
> <exclusions>
> <exclusion>
> <groupId>org.mortbay.jetty</groupId>
> <artifactId>servlet-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>javax.servlet</groupId>
> <artifactId>servlet-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>io.netty</groupId>
> <artifactId>netty</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
> to
>
> <dependency>
>
> <groupId>org.apache.hadoop</groupId>
>
> <artifactId>hadoop-client</artifactId>
>
> <version>1.0.4</version>
>
> <scope>provided</scope>
>
> <exclusions>
>
> <exclusion>
>
> <groupId>org.mortbay.jetty</groupId>
>
> <artifactId>servlet-api</artifactId>
>
> </exclusion>
>
> <exclusion>
>
> <groupId>javax.servlet</groupId>
>
> <artifactId>servlet-api</artifactId>
>
> </exclusion>
>
> <exclusion>
>
> <groupId>io.netty</groupId>
>
> <artifactId>netty</artifactId>
>
> </exclusion>
>
> </exclusions>
>
> </dependency>
>
>
> the warning message is still shown up in the namenode log. Is there any
> other thing I need to do?
>
>
> Thanks.
>
>
> Ey-Chih Chow
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
Thanks. But after I replace the maven dependence from
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.5.0-cdh5.2.0</version> <scope>provided</scope> <exclusions> <exclusion> <groupId>org.mortbay.jetty</groupId> <artifactId>servlet-api</artifactId> </exclusion> <exclusion> <groupId>javax.servlet</groupId> <artifactId>servlet-api</artifactId> </exclusion> <exclusion> <groupId>io.netty</groupId> <artifactId>netty</artifactId> </exclusion> </exclusions> </dependency>
to <dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>1.0.4</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>org.mortbay.jetty</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
the warning message is still shown up in the namenode log. Is there any other thing I need to do?
Thanks.
Ey-Chih Chow
> From: sowen@cloudera.com
> Date: Thu, 22 Jan 2015 22:34:22 +0000
> Subject: Re: spark 1.1.0 save data to hdfs failed
> To: eychih@hotmail.com
> CC: yuzhihong@gmail.com; user@spark.apache.org
>
> It means your client app is using Hadoop 2.x and your HDFS is Hadoop 1.x.
>
> On Thu, Jan 22, 2015 at 10:32 PM, ey-chih chow <ey...@hotmail.com> wrote:
> > I looked into the namenode log and found this message:
> >
> > 2015-01-22 22:18:39,441 WARN org.apache.hadoop.ipc.Server: Incorrect header
> > or version mismatch from 10.33.140.233:53776 got version 9 expected version
> > 4
> >
> > What should I do to fix this?
> >
> > Thanks.
> >
> > Ey-Chih
> >
> > ________________________________
> > From: eychih@hotmail.com
> > To: yuzhihong@gmail.com
> > CC: user@spark.apache.org
> > Subject: RE: spark 1.1.0 save data to hdfs failed
> > Date: Wed, 21 Jan 2015 23:12:56 -0800
> >
> > The hdfs release should be hadoop 1.0.4.
> >
> > Ey-Chih Chow
> >
> > ________________________________
> > Date: Wed, 21 Jan 2015 16:56:25 -0800
> > Subject: Re: spark 1.1.0 save data to hdfs failed
> > From: yuzhihong@gmail.com
> > To: eychih@hotmail.com
> > CC: user@spark.apache.org
> >
> > What hdfs release are you using ?
> >
> > Can you check namenode log around time of error below to see if there is
> > some clue ?
> >
> > Cheers
> >
> > On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
> >
> > Hi,
> >
> > I used the following fragment of a scala program to save data to hdfs:
> >
> > contextAwareEvents
> > .map(e => (new AvroKey(e), null))
> > .saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
> > + dateDir,
> > classOf[AvroKey[GenericRecord]],
> > classOf[NullWritable],
> > classOf[AvroKeyOutputFormat[GenericRecord]],
> > job.getConfiguration)
> >
> > But it failed with the following error messages. Is there any people who
> > can help? Thanks.
> >
> > Ey-Chih Chow
> >
> > =============================================
> >
> > Exception in thread "main" java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> > at
> > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> > Caused by: java.io.IOException: Failed on local exception:
> > java.io.EOFException; Host Details : local host is:
> > "ip-10-33-140-157/10.33.140.157"; destination host is:
> > "ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1415)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> > at
> > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> > at
> > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> > at
> > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> > at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
> > at
> > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> > at
> > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
> > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> > at
> > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
> > at
> > org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> > at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
> > at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> > ... 6 more
> > Caused by: java.io.EOFException
> > at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > at
> > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
> > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
> >
> > =======================================================
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
Re: spark 1.1.0 save data to hdfs failed
Posted by Sean Owen <so...@cloudera.com>.
It means your client app is using Hadoop 2.x and your HDFS is Hadoop 1.x.
On Thu, Jan 22, 2015 at 10:32 PM, ey-chih chow <ey...@hotmail.com> wrote:
> I looked into the namenode log and found this message:
>
> 2015-01-22 22:18:39,441 WARN org.apache.hadoop.ipc.Server: Incorrect header
> or version mismatch from 10.33.140.233:53776 got version 9 expected version
> 4
>
> What should I do to fix this?
>
> Thanks.
>
> Ey-Chih
>
> ________________________________
> From: eychih@hotmail.com
> To: yuzhihong@gmail.com
> CC: user@spark.apache.org
> Subject: RE: spark 1.1.0 save data to hdfs failed
> Date: Wed, 21 Jan 2015 23:12:56 -0800
>
> The hdfs release should be hadoop 1.0.4.
>
> Ey-Chih Chow
>
> ________________________________
> Date: Wed, 21 Jan 2015 16:56:25 -0800
> Subject: Re: spark 1.1.0 save data to hdfs failed
> From: yuzhihong@gmail.com
> To: eychih@hotmail.com
> CC: user@spark.apache.org
>
> What hdfs release are you using ?
>
> Can you check namenode log around time of error below to see if there is
> some clue ?
>
> Cheers
>
> On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
>
> Hi,
>
> I used the following fragment of a scala program to save data to hdfs:
>
> contextAwareEvents
> .map(e => (new AvroKey(e), null))
> .saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
> + dateDir,
> classOf[AvroKey[GenericRecord]],
> classOf[NullWritable],
> classOf[AvroKeyOutputFormat[GenericRecord]],
> job.getConfiguration)
>
> But it failed with the following error messages. Is there any people who
> can help? Thanks.
>
> Ey-Chih Chow
>
> =============================================
>
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> at
> org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.io.IOException: Failed on local exception:
> java.io.EOFException; Host Details : local host is:
> "ip-10-33-140-157/10.33.140.157"; destination host is:
> "ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1415)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
> at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> ... 6 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
>
> =======================================================
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
I looked into the namenode log and found this message:
2015-01-22 22:18:39,441 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 10.33.140.233:53776 got version 9 expected version 4
What should I do to fix this?
Thanks.
Ey-Chih
From: eychih@hotmail.com
To: yuzhihong@gmail.com
CC: user@spark.apache.org
Subject: RE: spark 1.1.0 save data to hdfs failed
Date: Wed, 21 Jan 2015 23:12:56 -0800
The hdfs release should be hadoop 1.0.4.
Ey-Chih Chow
Date: Wed, 21 Jan 2015 16:56:25 -0800
Subject: Re: spark 1.1.0 save data to hdfs failed
From: yuzhihong@gmail.com
To: eychih@hotmail.com
CC: user@spark.apache.org
What hdfs release are you using ?
Can you check namenode log around time of error below to see if there is some clue ?
Cheers
On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
Hi,
I used the following fragment of a scala program to save data to hdfs:
contextAwareEvents
.map(e => (new AvroKey(e), null))
.saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
+ dateDir,
classOf[AvroKey[GenericRecord]],
classOf[NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
job.getConfiguration)
But it failed with the following error messages. Is there any people who
can help? Thanks.
Ey-Chih Chow
=============================================
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.io.IOException: Failed on local exception:
java.io.EOFException; Host Details : local host is:
"ip-10-33-140-157/10.33.140.157"; destination host is:
"ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
... 6 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
=======================================================
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
RE: spark 1.1.0 save data to hdfs failed
Posted by ey-chih chow <ey...@hotmail.com>.
The hdfs release should be hadoop 1.0.4.
Ey-Chih Chow
Date: Wed, 21 Jan 2015 16:56:25 -0800
Subject: Re: spark 1.1.0 save data to hdfs failed
From: yuzhihong@gmail.com
To: eychih@hotmail.com
CC: user@spark.apache.org
What hdfs release are you using ?
Can you check namenode log around time of error below to see if there is some clue ?
Cheers
On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
Hi,
I used the following fragment of a scala program to save data to hdfs:
contextAwareEvents
.map(e => (new AvroKey(e), null))
.saveAsNewAPIHadoopFile("hdfs://" + masterHostname + ":9000/ETL/output/"
+ dateDir,
classOf[AvroKey[GenericRecord]],
classOf[NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
job.getConfiguration)
But it failed with the following error messages. Is there any people who
can help? Thanks.
Ey-Chih Chow
=============================================
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.io.IOException: Failed on local exception:
java.io.EOFException; Host Details : local host is:
"ip-10-33-140-157/10.33.140.157"; destination host is:
"ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
... 6 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
=======================================================
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark 1.1.0 save data to hdfs failed
Posted by Ted Yu <yu...@gmail.com>.
What hdfs release are you using ?
Can you check namenode log around time of error below to see if there is
some clue ?
Cheers
On Wed, Jan 21, 2015 at 4:51 PM, ey-chih chow <ey...@hotmail.com> wrote:
> Hi,
>
> I used the following fragment of a scala program to save data to hdfs:
>
> contextAwareEvents
> .map(e => (new AvroKey(e), null))
> .saveAsNewAPIHadoopFile("hdfs://" + masterHostname +
> ":9000/ETL/output/"
> + dateDir,
> classOf[AvroKey[GenericRecord]],
> classOf[NullWritable],
> classOf[AvroKeyOutputFormat[GenericRecord]],
> job.getConfiguration)
>
> But it failed with the following error messages. Is there any people who
> can help? Thanks.
>
> Ey-Chih Chow
>
> =============================================
>
> Exception in thread "main" java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
> at
> org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
> Caused by: java.io.IOException: Failed on local exception:
> java.io.EOFException; Host Details : local host is:
> "ip-10-33-140-157/10.33.140.157"; destination host is:
> "ec2-54-203-58-2.us-west-2.compute.amazonaws.com":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1415)
> at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy14.getFileInfo(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:744)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1925)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1079)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1075)
> at
>
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1075)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> at
>
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> at
>
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:900)
> at
>
> org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:832)
> at com.crowdstar.etl.ParseAndClean$.main(ParseAndClean.scala:101)
> at com.crowdstar.etl.ParseAndClean.main(ParseAndClean.scala)
> ... 6 more
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
>
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950)
>
> =======================================================
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-1-0-save-data-to-hdfs-failed-tp21305.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>