You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Michel Hubert <mi...@phact.nl> on 2016/03/22 10:13:00 UTC

Spark 1.6.0 op CDH 5.6.0

Hi,


I'm trying to run a Spark 1.6.0 application on a CDH 5.6.0 cluster.

How do I submit the uber-jar so it's totally self-reliant?

With kind regards,
Mitchel


spark-submit --class TEST --master yarn-cluster ./uber-TEST-1.0-SNAPSHOT.jar


Spark 1.6.1
Version: Cloudera Express 5.6.0

16/03/22 09:16:33 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.streaming.api.java.JavaPairDStream.mapWithState(Lorg/apache/spark/streaming/StateSpec;)Lorg/apache/spark/streaming/api/java/JavaMapWithStateDStream;)

Re: Spark 1.6.0 op CDH 5.6.0

Posted by Sebastian YEPES FERNANDEZ <sy...@gmail.com>.

Hello Michel,

I had a similar issue when running my custom built Spark 1.6.1, at the end
I resolved the issue by building Spark and my Jar with the CDH built in jvm

export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/


Regards and hope this helps,
Sebastian

On Tue, Mar 22, 2016 at 10:13 AM, Michel Hubert <mi...@phact.nl> wrote:

> Hi,
>
>
>
>
>
> I’m trying to run a Spark 1.6.0 application on a CDH 5.6.0 cluster.
>
>
>
> How do I submit the uber-jar so it’s totally self-reliant?
>
>
>
> With kind regards,
>
> 
> Mitchel
>
>
>
>
>
> spark-submit --class TEST --master yarn-cluster
> ./uber-TEST-1.0-SNAPSHOT.jar
>
>
>
>
>
> Spark 1.6.1
>
> *Version*: Cloudera Express 5.6.0
>
>
>
> 16/03/22 09:16:33 INFO yarn.ApplicationMaster: Unregistering
> ApplicationMaster with FAILED (diag message: User class threw exception:
> java.lang.NoSuchMethodError:
> org.apache.spark.streaming.api.java.JavaPairDStream.mapWithState(Lorg/apache/spark/streaming/StateSpec;)Lorg/apache/spark/streaming/api/java/JavaMapWithStateDStream;)
>
>
>
>
>
>
>