You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by M Singh <ma...@yahoo.com> on 2014/06/27 21:58:30 UTC
jackson-core-asl jar (1.8.8 vs 1.9.x) conflict with the spark-sql (version 1.x)
Hi:
I am using spark to stream data to cassandra and it works fine in local mode. But when I execute the application in a standalone clustered env I got exception included below (java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass).
I think this is due to the jackson-core-asl dependency conflict (jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not). The 1.9.x version is being pulled in by spark-sql project. I tried adding jackson-core-asl 1.8.8 with --jars argument while submitting the application for execution but it did not work. So I created a custom spark build excluding sql project. With this custom spark install I was able to resolve the issue at least on a single node cluster (separate master and worker).
If there is an alternate way to resolve this conflicting jar issue without a custom build (eg: configuration to use the user defined jars in the executor class path first), please let me know.
Also, is there a comprehensive list of configuration properties available for spark ?
Thanks
Mans
Exception trace
TaskSetManager: Loss was due to java.lang.NoClassDefFoundError
java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
at org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
at org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
at org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
at org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602)
Re: jackson-core-asl jar (1.8.8 vs 1.9.x) conflict with the spark-sql (version 1.x)
Posted by Gino Bustelo <lb...@gmail.com>.
Hit this same problem yesterday. My fix might not be ideal for you, but we were able to get rid of the error by turning off annotation deser in ObjectMapper.
Gino B.
> On Jun 27, 2014, at 2:58 PM, M Singh <ma...@yahoo.com> wrote:
>
> Hi:
>
> I am using spark to stream data to cassandra and it works fine in local mode. But when I execute the application in a standalone clustered env I got exception included below (java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass).
>
> I think this is due to the jackson-core-asl dependency conflict (jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not). The 1.9.x version is being pulled in by spark-sql project. I tried adding jackson-core-asl 1.8.8 with --jars argument while submitting the application for execution but it did not work. So I created a custom spark build excluding sql project. With this custom spark install I was able to resolve the issue at least on a single node cluster (separate master and worker).
>
> If there is an alternate way to resolve this conflicting jar issue without a custom build (eg: configuration to use the user defined jars in the executor class path first), please let me know.
>
> Also, is there a comprehensive list of configuration properties available for spark ?
>
> Thanks
>
> Mans
>
> Exception trace
>
> TaskSetManager: Loss was due to java.lang.NoClassDefFoundError
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
> at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
> at org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
> at org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229)
> at org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386)
> at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
> at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
> at org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
> at org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
> at org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
> at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602)
>
Re: jackson-core-asl jar (1.8.8 vs 1.9.x) conflict with the spark-sql (version 1.x)
Posted by M Singh <ma...@yahoo.com>.
Hi Paul:
Here are the dependencies in spark 1.1.0-snapshot that are pulling in org.codehaus.jackson:jackson-core-asl 1.8 and 1.9 jar.
1.9
com.twitter:parquet-hadoop:jar:1.4.3
org.apache.avro:avro:jar:1.7.6
1.8
org.apache.spark:spark-hive_2.10:jar:1.1.0-SNAPSHOT
org.apache.hadoop:hadoop-core:jar:1.0.4
org.apache.hbase:hbase:jar:0.94.6
Thanks
Mans
On Saturday, June 28, 2014 2:22 AM, Paul Brown <pr...@mult.ifario.us> wrote:
Hi, Mans --
Both of those versions of Jackson are pretty ancient. Do you know which of the Spark dependencies is pulling them in? It would be good for us (the Jackson, Woodstox, etc., folks) to see if we can get people to upgrade to more recent versions of Jackson.
-- Paul
—
prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
On Fri, Jun 27, 2014 at 12:58 PM, M Singh <ma...@yahoo.com> wrote:
Hi:
>
>
>I am using spark to stream data to cassandra and it works fine in local mode. But when I execute the application in a standalone clustered env I got exception included below (java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass).
>
>
>I think this is due to the jackson-core-asl dependency conflict (jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not). The 1.9.x version is being pulled in by spark-sql project. I tried adding jackson-core-asl 1.8.8 with --jars argument while submitting the application for execution but it did not work. So I created a custom spark build excluding sql project. With this custom spark install I was able to resolve the issue at least on a single node cluster (separate master and worker).
>
>If there is an alternate way to resolve this conflicting jar issue without a custom build (eg: configuration to use the user defined jars in the executor class path first), please let me know.
>
>Also, is there a comprehensive list of configuration properties available for spark ?
>
>
>
>Thanks
>
>
>Mans
>
>
>Exception trace
>
>
> TaskSetManager: Loss was due to java.lang.NoClassDefFoundError
>java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
>at org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
>at org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
>at
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229)
>at org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386)
>at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
>at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
>at org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
>at org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
>at org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
>at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
>at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602)
>
>
Re: jackson-core-asl jar (1.8.8 vs 1.9.x) conflict with the spark-sql
(version 1.x)
Posted by Paul Brown <pr...@mult.ifario.us>.
Hi, Mans --
Both of those versions of Jackson are pretty ancient. Do you know which of
the Spark dependencies is pulling them in? It would be good for us (the
Jackson, Woodstox, etc., folks) to see if we can get people to upgrade to
more recent versions of Jackson.
-- Paul
—
prb@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/
On Fri, Jun 27, 2014 at 12:58 PM, M Singh <ma...@yahoo.com> wrote:
> Hi:
>
> I am using spark to stream data to cassandra and it works fine in local
> mode. But when I execute the application in a standalone clustered env I
> got exception included below (java.lang.NoClassDefFoundError:
> org/codehaus/jackson/annotate/JsonClass).
>
> I think this is due to the jackson-core-asl dependency conflict
> (jackson-core-asl 1.8.8 has the JsonClass but 1.9.x does not). The 1.9.x
> version is being pulled in by spark-sql project. I tried adding
> jackson-core-asl 1.8.8 with --jars argument while submitting the
> application for execution but it did not work. So I created a custom spark
> build excluding sql project. With this custom spark install I was able to
> resolve the issue at least on a single node cluster (separate master and
> worker).
>
> If there is an alternate way to resolve this conflicting jar issue without
> a custom build (eg: configuration to use the user defined jars in the
> executor class path first), please let me know.
>
> Also, is there a comprehensive list of configuration properties available
> for spark ?
>
> Thanks
>
> Mans
>
> Exception trace
>
> TaskSetManager: Loss was due to java.lang.NoClassDefFoundError
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
> at
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:229)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:386)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
> at
> org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1602)
>
>