You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sandy Ryza <sa...@cloudera.com> on 2014/01/07 05:51:32 UTC

NoSuchMethodError running Spark on YARN

Hi,

I hit this when trying to an example job with Spark master against YARN:

Exception in thread "main" java.lang.NoSuchMethodError:
com.typesafe.config.ConfigFactory.invalidateCaches()V
at org.apache.spark.SparkConf.<init>(SparkConf.scala:35)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:30)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:487)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)

I pretty much followed the 0.8.1 doc:
$MY_SPARK_HOME/bin/spark-class org.apache.spark.deploy.yarn.Client --jar
$MY_SPARK_HOME/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating.jar
--class
org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers
1 --master-memory 1g --worker-memory 1g --worker-cores 1

Any idea what's going on?  Will keep digging, but thought somebody might
recognize it.

thanks for any guidance,
Sandy

Re: NoSuchMethodError running Spark on YARN

Posted by Sean Owen <sr...@gmail.com>.

Sandy looks like a version mismatch: compiling vs Spark HEAD and
running vs 0.8.1?

The method was added to typesafe config on Oct 8 2012 and released in
about version 0.6:
https://github.com/typesafehub/config/commit/5f486f65ac68745ca89059a5b6b144c2daa5d157#diff-3d5ac6ed49837be68d4f47d3b96b1c81

typesafe config comes in via Akka, and Spark 0.8.1 depends on
akka-actor 2.0.5 which in turn depends on config 0.3.1:
[INFO] +- org.apache.spark:spark-core_2.9.3:jar:0.8.1-incubating:compile
...
[INFO] |  +- com.typesafe.akka:akka-actor:jar:2.0.5:compile
[INFO] |  |  \- com.typesafe:config:jar:0.3.1:compile

http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-parent%7C0.8.1-incubating%7Cpom

But Spark 0.9.0-SNAPSHOT / HEAD depends on akka-actor 2.2.3 which
depends on typesafe config 1.0.2.

On Tue, Jan 7, 2014 at 5:01 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> Verified that com.typesafe.config.ConfigFactory is present in the Spark
> assembly jar so now I'm even more confused.
>
>
> On Mon, Jan 6, 2014 at 8:51 PM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> Hi,
>>
>> I hit this when trying to an example job with Spark master against YARN:
>>
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> com.typesafe.config.ConfigFactory.invalidateCaches()V
>> at org.apache.spark.SparkConf.<init>(SparkConf.scala:35)
>> at org.apache.spark.SparkConf.<init>(SparkConf.scala:30)
>> at org.apache.spark.deploy.yarn.Client$.main(Client.scala:487)
>> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
>>
>> I pretty much followed the 0.8.1 doc:
>> $MY_SPARK_HOME/bin/spark-class org.apache.spark.deploy.yarn.Client --jar
>> $MY_SPARK_HOME/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating.jar
>> --class org.apache.spark.examples.SparkPi --args yarn-standalone
>> --num-workers 1 --master-memory 1g --worker-memory 1g --worker-cores 1
>>
>> Any idea what's going on?  Will keep digging, but thought somebody might
>> recognize it.
>>
>> thanks for any guidance,
>> Sandy
>
>

Re: NoSuchMethodError running Spark on YARN

Posted by Sandy Ryza <sa...@cloudera.com>.

Verified that com.typesafe.config.ConfigFactory is present in the Spark
assembly jar so now I'm even more confused.


On Mon, Jan 6, 2014 at 8:51 PM, Sandy Ryza <sa...@cloudera.com> wrote:

> Hi,
>
> I hit this when trying to an example job with Spark master against YARN:
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> com.typesafe.config.ConfigFactory.invalidateCaches()V
>  at org.apache.spark.SparkConf.<init>(SparkConf.scala:35)
> at org.apache.spark.SparkConf.<init>(SparkConf.scala:30)
>  at org.apache.spark.deploy.yarn.Client$.main(Client.scala:487)
> at org.apache.spark.deploy.yarn.Client.main(Client.scala)
>
> I pretty much followed the 0.8.1 doc:
> $MY_SPARK_HOME/bin/spark-class org.apache.spark.deploy.yarn.Client --jar
> $MY_SPARK_HOME/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating.jar --class
> org.apache.spark.examples.SparkPi --args yarn-standalone --num-workers
> 1 --master-memory 1g --worker-memory 1g --worker-cores 1
>
> Any idea what's going on?  Will keep digging, but thought somebody might
> recognize it.
>
> thanks for any guidance,
> Sandy
>