You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andy Davidson <An...@SantaCruzIntegration.com> on 2014/09/18 01:28:27 UTC

spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass

Hi I am new to spark.

I am trying to write a simple java program that process tweets that where
collected and stored in a file. I figured the simplest thing to do would be
to convert the JSON string into a java map. When I submit my jar file I keep
getting the following error

java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass


For the life of me I can not figure out what the problem is. I am using
maven shade plugin and check to see that the missing class is my uber jar

Any suggestions would be greatly appreciated.

Andy

P.s. I should mention that I am running everything on my local machine.

<properties>

<jackson.version>1.8.8</jackson.version> <!-- 1.9.13 -->

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

</properties>



<dependencies>

<dependency>

<groupId>org.codehaus.jackson</groupId>

<artifactId>jackson-core-asl</artifactId>

<version>${jackson.version}</version>

</dependency>

<dependency>

<groupId>org.codehaus.jackson</groupId>

<artifactId>jackson-mapper-asl</artifactId>

<version>${jackson.version}</version>

</dependency>



I am using shade to build an uber jar



$jar ­tvf target/myUber.jar

Š

   580 Wed Sep 17 16:17:36 PDT 2014
org/codehaus/jackson/annotate/JsonClass.class

Š




14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)

java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass

at 
org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeseri
alizationType(JacksonAnnotationIntrospector.java:524)

at 
org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotati
on(BasicDeserializerFactory.java:732)

at 
org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserialize
r(BasicDeserializerFactory.java:337)

at 
org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(S
tdDeserializerProvider.java:377)

at 
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdD
eserializerProvider.java:307)

at 
org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueD
eserializer(StdDeserializerProvider.java:287)

at 
org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer
(StdDeserializerProvider.java:136)

at 
org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeseria
lizer(StdDeserializerProvider.java:157)

at 
org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.jav
a:2468)

at 
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:240
2)

at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)

at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)





public class JSONHelper {

    public static final ObjectMapper mapper = new ObjectMapper();

    static {

        mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);

        //mapper.configure(Feature.USE_ANNOTATIONS, false);

    }

Š

   public static Map<String, String> toJsonMap(String json) {

        System.err.println("AEDWIP toJsonMap: " + json);

        try {

            TypeReference<HashMap<String, String>> typeRef = new
TypeReference<HashMap<String, String>>() {

            };

            //return mapper.readValue(json, new
TypeReference<HashMap<String,String>>(){});

            return mapper.readValue(json, typeRef);

            //return mapper.convertValue(json, typeRef);

            

//            HashMap<String,String> ret =

//                    new ObjectMapper().readValue(json, HashMap.class);

        } catch (Exception e) {

            throw new IOError(e);

        }

    }

Re: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass

Posted by Yin Huai <hu...@gmail.com>.
Hello Andy,

Will our JSON support in Spark SQL help your case? If your JSON files store
one JSON object per line, you can use SQLContext.jsonFile to load it. If
you want to do pre-process these files, once you have an RDD[String] (one
JSON object per String), you can use SQLContext.jsonRDD. In both ways,
Spark SQL will infer the schema and you can query your datasets like
querying tables. The instruction about using jsonFile and jsonRDD can be
found at
https://spark.apache.org/docs/latest/sql-programming-guide.html#json-datasets
.

Thanks,

Yin

On Thu, Sep 18, 2014 at 2:36 PM, Andy Davidson <
Andy@santacruzintegration.com> wrote:

> After lots of hacking I figure out how to resolve this problem. This is
> good solution. It severalty cripples jackson but at least for now I am
> unblocked
>
> 1) turn off annotations.
>         mapper.configure(Feature.USE_ANNOTATIONS, false);
>
>
> 2) in maven set the jackson dependencies as provided.
>
> <properties>
>
> <jackson.version>1.9.13</jackson.version> <!-- 1.8.8 1.9.13 2.3.4-Final
> -->
>
> </properties>
>
>
>  <dependencies>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-core-asl</artifactId>
>
> <scope>provided</scope>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-mapper-asl</artifactId>
>
> <scope>provided</scope>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
>
> Hope this helps someone else
>
>
> andy
>
>
> From: Andrew Davidson <An...@SantaCruzIntegration.com>
> Date: Wednesday, September 17, 2014 at 4:28 PM
> To: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError:
> org/codehaus/jackson/annotate/JsonClass
>
> Hi I am new to spark.
>
> I am trying to write a simple java program that process tweets that where
> collected and stored in a file. I figured the simplest thing to do would be
> to convert the JSON string into a java map. When I submit my jar file I
> keep getting the following error
>
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
>
> For the life of me I can not figure out what the problem is. I am using
> maven shade plugin and check to see that the missing class is my uber jar
>
> Any suggestions would be greatly appreciated.
>
> Andy
>
> P.s. I should mention that I am running everything on my local machine.
>
> <properties>
>
> <jackson.version>1.8.8</jackson.version> <!-- 1.9.13 -->
>
> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>
> </properties>
>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-core-asl</artifactId>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>org.codehaus.jackson</groupId>
>
> <artifactId>jackson-mapper-asl</artifactId>
>
> <version>${jackson.version}</version>
>
> </dependency>
>
>
> I am using shade to build an uber jar
>
>
> $jar –tvf target/myUber.jar
>
> …
>
>    580 Wed Sep 17 16:17:36 PDT 2014
> org/codehaus/jackson/annotate/JsonClass.class
> …
>
>
> 14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
> 0)
>
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
>
> at
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserializationType(JacksonAnnotationIntrospector.java:524)
>
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation(BasicDeserializerFactory.java:732)
>
> at
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserializer(BasicDeserializerFactory.java:337)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:377)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:307)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:287)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:136)
>
> at
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:157)
>
> at
> org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2468)
>
> at
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
>
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
>
> at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
>
>
>
> public class JSONHelper {
>
>     public static final ObjectMapper mapper = new ObjectMapper();
>
>     static {
>
>         mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
>
>         //mapper.configure(Feature.USE_ANNOTATIONS, false);
>
>     }
>
> …
>
>    public static Map<String, String> toJsonMap(String json) {
>
>         System.err.println("AEDWIP toJsonMap: " + json);
>
>         try {
>
>             TypeReference<HashMap<String, String>> typeRef = new
> TypeReference<HashMap<String, String>>() {
>
>             };
>
>             //return mapper.readValue(json, new
> TypeReference<HashMap<String,String>>(){});
>
>             return mapper.readValue(json, typeRef);
>
>             //return mapper.convertValue(json, typeRef);
>
>
>
> //            HashMap<String,String> ret =
>
> //                    new ObjectMapper().readValue(json, HashMap.class);
>
>         } catch (Exception e) {
>
>             throw new IOError(e);
>
>         }
>
>     }
>
>
>
>
>
>

Re: spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass

Posted by Andy Davidson <An...@SantaCruzIntegration.com>.
After lots of hacking I figure out how to resolve this problem. This is good
solution. It severalty cripples jackson but at least for now I am unblocked

1) turn off annotations.
        mapper.configure(Feature.USE_ANNOTATIONS, false);


2) in maven set the jackson dependencies as provided.

<properties>

<jackson.version>1.9.13</jackson.version> <!-- 1.8.8 1.9.13 2.3.4-Final -->

</properties>



<dependencies>

<dependency>

<groupId>org.codehaus.jackson</groupId>

<artifactId>jackson-core-asl</artifactId>

<scope>provided</scope>

<version>${jackson.version}</version>

</dependency>

<dependency>

<groupId>org.codehaus.jackson</groupId>

<artifactId>jackson-mapper-asl</artifactId>

<scope>provided</scope>

<version>${jackson.version}</version>

</dependency>



Hope this helps someone else



andy


From:  Andrew Davidson <An...@SantaCruzIntegration.com>
Date:  Wednesday, September 17, 2014 at 4:28 PM
To:  "user@spark.apache.org" <us...@spark.apache.org>
Subject:  spark-1.1.0-bin-hadoop2.4 java.lang.NoClassDefFoundError:
org/codehaus/jackson/annotate/JsonClass

> Hi I am new to spark.
> 
> I am trying to write a simple java program that process tweets that where
> collected and stored in a file. I figured the simplest thing to do would be to
> convert the JSON string into a java map. When I submit my jar file I keep
> getting the following error
> 
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
> 
> 
> For the life of me I can not figure out what the problem is. I am using maven
> shade plugin and check to see that the missing class is my uber jar
> 
> Any suggestions would be greatly appreciated.
> 
> Andy
> 
> P.s. I should mention that I am running everything on my local machine.
> 
> <properties>
> 
> <jackson.version>1.8.8</jackson.version> <!-- 1.9.13 -->
> 
> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
> 
> </properties>
> 
> 
> 
> <dependencies>
> 
> <dependency>
> 
> <groupId>org.codehaus.jackson</groupId>
> 
> <artifactId>jackson-core-asl</artifactId>
> 
> <version>${jackson.version}</version>
> 
> </dependency>
> 
> <dependency>
> 
> <groupId>org.codehaus.jackson</groupId>
> 
> <artifactId>jackson-mapper-asl</artifactId>
> 
> <version>${jackson.version}</version>
> 
> </dependency>
> 
> 
> 
> I am using shade to build an uber jar
> 
> 
> 
> $jar ­tvf target/myUber.jar
> 
> Š
> 
>    580 Wed Sep 17 16:17:36 PDT 2014
> org/codehaus/jackson/annotate/JsonClass.class
> 
> Š
> 
> 
> 
> 
> 14/09/17 16:13:24 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> 
> java.lang.NoClassDefFoundError: org/codehaus/jackson/annotate/JsonClass
> 
> at 
> org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector.findDeserial
> izationType(JacksonAnnotationIntrospector.java:524)
> 
> at 
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.modifyTypeByAnnotation
> (BasicDeserializerFactory.java:732)
> 
> at 
> org.codehaus.jackson.map.deser.BasicDeserializerFactory.createMapDeserializer(
> BasicDeserializerFactory.java:337)
> 
> at 
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(Std
> DeserializerProvider.java:377)
> 
> at 
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDes
> erializerProvider.java:307)
> 
> at 
> org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDes
> erializer(StdDeserializerProvider.java:287)
> 
> at 
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(S
> tdDeserializerProvider.java:136)
> 
> at 
> org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeseriali
> zer(StdDeserializerProvider.java:157)
> 
> at 
> org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:
> 2468)
> 
> at 
> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2402)
> 
> at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1609)
> 
> at com.santacruzintegration.spark.JSONHelper.toJsonMap(JSONHelper.java:40)
> 
> 
> 
> 
> 
> public class JSONHelper {
> 
>     public static final ObjectMapper mapper = new ObjectMapper();
> 
>     static {
> 
>         mapper.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
> 
>         //mapper.configure(Feature.USE_ANNOTATIONS, false);
> 
>     }
> 
> Š
> 
>    public static Map<String, String> toJsonMap(String json) {
> 
>         System.err.println("AEDWIP toJsonMap: " + json);
> 
>         try {
> 
>             TypeReference<HashMap<String, String>> typeRef = new
> TypeReference<HashMap<String, String>>() {
> 
>             };
> 
>             //return mapper.readValue(json, new
> TypeReference<HashMap<String,String>>(){});
> 
>             return mapper.readValue(json, typeRef);
> 
>             //return mapper.convertValue(json, typeRef);
> 
>             
> 
> //            HashMap<String,String> ret =
> 
> //                    new ObjectMapper().readValue(json, HashMap.class);
> 
>         } catch (Exception e) {
> 
>             throw new IOError(e);
> 
>         }
> 
>     }
> 
>     
> 
> 
>