You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Charlie Wright <ch...@live.ca> on 2016/02/12 02:15:45 UTC

Building Spark with a Custom Version of Hadoop: HDFS ClassNotFoundException

I am having issues trying to run a test job on a built version of Spark with a custom Hadoop JAR. My custom hadoop version runs without issues and I can run jobs from a precompiled version of Spark (with Hadoop) no problem. 
However, whenever I try to run the same Spark example on the Spark version with my custom hadoop JAR - I get this error:"Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.Hdfs not found"
Does anybody know why this is happening?
Thanks,Charles.

Re: Building Spark with a Custom Version of Hadoop: HDFS ClassNotFoundException

Posted by Ted Yu <yu...@gmail.com>.
The Spark driver does not run on the YARN cluster in client mode, only the
Spark executors do.
Can you check YARN logs for the failed job to see if there was more clue ?

Does the YARN cluster run the customized hadoop or stock hadoop ?

Cheers

On Thu, Feb 11, 2016 at 5:44 PM, Charlie Wright <ch...@live.ca>
wrote:

> ./bin/spark-submit --class org.apache.spark.examples.SparkPi     --master
> yarn     --deploy-mode client     --driver-memory 4g     --executor-memory
> 1664m     --executor-cores 1     --queue default
> examples/target/spark-examples*.jar     10
>
> I am using the 1.6.0 release.
>
>
> Charles.
>
> ------------------------------
> Date: Thu, 11 Feb 2016 17:41:54 -0800
> Subject: Re: Building Spark with a Custom Version of Hadoop: HDFS
> ClassNotFoundException
> From: yuzhihong@gmail.com
> To: charliewright@live.ca; user@spark.apache.org
>
>
> I think SPARK_CLASSPATH is deprecated.
>
> Can you show the command line launching your Spark job ?
> Which Spark release do you use ?
>
> Thanks
>
>
>
> On Thu, Feb 11, 2016 at 5:38 PM, Charlie Wright <ch...@live.ca>
> wrote:
>
> built and installed hadoop with:
> mvn package -Pdist -DskipTests -Dtar
> mvn install -DskipTests
>
> built spark with:
> mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.8.0-SNAPSHOT -DskipTests clean
> package
>
> Where would I check the classpath? Is it the environment variable
> SPARK_CLASSPATH?
>
> Charles
>
> ------------------------------
> Date: Thu, 11 Feb 2016 17:29:00 -0800
> Subject: Re: Building Spark with a Custom Version of Hadoop: HDFS
> ClassNotFoundException
> From: yuzhihong@gmail.com
> To: charliewright@live.ca
> CC: dev@spark.apache.org
>
> Hdfs class is in hadoop-hdfs-XX.jar
>
> Can you check the classpath to see if the above jar is there ?
>
> Please describe the command lines you used for building hadoop / Spark.
>
> Cheers
>
> On Thu, Feb 11, 2016 at 5:15 PM, Charlie Wright <ch...@live.ca>
> wrote:
>
> I am having issues trying to run a test job on a built version of Spark
> with a custom Hadoop JAR.
> My custom hadoop version runs without issues and I can run jobs from a
> precompiled version of Spark (with Hadoop) no problem.
>
> However, whenever I try to run the same Spark example on the Spark version
> with my custom hadoop JAR - I get this error:
> "Exception in thread "main" java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.Hdfs not found"
>
> Does anybody know why this is happening?
>
> Thanks,
> Charles.
>
>
>
>

Re: Building Spark with a Custom Version of Hadoop: HDFS ClassNotFoundException

Posted by Ted Yu <yu...@gmail.com>.
I think SPARK_CLASSPATH is deprecated.

Can you show the command line launching your Spark job ?
Which Spark release do you use ?

Thanks



On Thu, Feb 11, 2016 at 5:38 PM, Charlie Wright <ch...@live.ca>
wrote:

> built and installed hadoop with:
> mvn package -Pdist -DskipTests -Dtar
> mvn install -DskipTests
>
> built spark with:
> mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.8.0-SNAPSHOT -DskipTests clean
> package
>
> Where would I check the classpath? Is it the environment variable
> SPARK_CLASSPATH?
>
> Charles
>
> ------------------------------
> Date: Thu, 11 Feb 2016 17:29:00 -0800
> Subject: Re: Building Spark with a Custom Version of Hadoop: HDFS
> ClassNotFoundException
> From: yuzhihong@gmail.com
> To: charliewright@live.ca
> CC: dev@spark.apache.org
>
> Hdfs class is in hadoop-hdfs-XX.jar
>
> Can you check the classpath to see if the above jar is there ?
>
> Please describe the command lines you used for building hadoop / Spark.
>
> Cheers
>
> On Thu, Feb 11, 2016 at 5:15 PM, Charlie Wright <ch...@live.ca>
> wrote:
>
> I am having issues trying to run a test job on a built version of Spark
> with a custom Hadoop JAR.
> My custom hadoop version runs without issues and I can run jobs from a
> precompiled version of Spark (with Hadoop) no problem.
>
> However, whenever I try to run the same Spark example on the Spark version
> with my custom hadoop JAR - I get this error:
> "Exception in thread "main" java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.Hdfs not found"
>
> Does anybody know why this is happening?
>
> Thanks,
> Charles.
>
>
>

Re: Building Spark with a Custom Version of Hadoop: HDFS ClassNotFoundException

Posted by Ted Yu <yu...@gmail.com>.
Hdfs class is in hadoop-hdfs-XX.jar

Can you check the classpath to see if the above jar is there ?

Please describe the command lines you used for building hadoop / Spark.

Cheers

On Thu, Feb 11, 2016 at 5:15 PM, Charlie Wright <ch...@live.ca>
wrote:

> I am having issues trying to run a test job on a built version of Spark
> with a custom Hadoop JAR.
> My custom hadoop version runs without issues and I can run jobs from a
> precompiled version of Spark (with Hadoop) no problem.
>
> However, whenever I try to run the same Spark example on the Spark version
> with my custom hadoop JAR - I get this error:
> "Exception in thread "main" java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.Hdfs not found"
>
> Does anybody know why this is happening?
>
> Thanks,
> Charles.
>
>