You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Christophe Billiard <ch...@gmail.com> on 2014/12/31 12:08:17 UTC
NoSuchMethodError: com.typesafe.config.Config.getDuration with
akka-http/akka-stream
Hi all,
I am currently trying to combine datastax's "spark-cassandra-connector" and
typesafe's "akka-http-experimental"
on Spark 1.1.1 (spark-cassandra-connector for Spark 1.2.0 not out yet) and
scala 2.10.4
I am using the hadoop 2.4 pre built package. (build.sbt file at the end)
To solve the java.lang.NoClassDefFoundError:
com/datastax/spark/connector/mapper/ColumnMapper
and other NoClassDefFoundErrors, I have to give some jars to Spark
(build.sbt is not enough).
The connectors works fine.
My spark submit looks like:
sbt clean package; bin/spark-submit --class "SimpleAppStreaming3"
--master local[*] --jars
"spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar"
target/scala-2.10/simple-project_2.10-1.0.jar
Then I am trying to add some akka-http/akka-stream features.
Like before I get a java.lang.NoClassDefFoundError:
akka/stream/FlowMaterializer$
Same solution, I begin to add jars.
Now my spark submit looks like:
sbt clean package; bin/spark-submit --class impleAppStreaming3" --master
local[*] --jars
"spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar","akka-stream-experimental_2.10-1.0-M2.jar"
target/scala-2.10/simple-project_2.10-1.0.jar
Then I have a new kind of error:
Exception in thread "main" java.lang.NoSuchMethodError:
com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
at
akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
at akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:185)
at akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:172)
at akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
at akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
at scala.Option.getOrElse(Option.scala:120)
at akka.stream.FlowMaterializer$.apply(FlowMaterializer.scala:42)
at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:240)
at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I can't get rid of this error.
I tried:
1) adding several jars (including "config-1.2.1.jar")
2) studying the dependency tree (with
https://github.com/jrudolph/sbt-dependency-graph)
3) excluding libraryDependencies (with dependencyOverrides)
Any ideas?
Bonus question: Is there a way to avoid adding all these jars with --jars?
*My build.sbt file*
name := "Simple Project"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
//exclude("com.typesafe", "config")
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" %
"2.1.3"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" %
"1.1.0" withSources() withJavadoc()
libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" % "2.0.5"
libraryDependencies += "joda-time" % "joda-time" % "2.6"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.8"
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.3.8"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.4.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.2"
libraryDependencies += "org.mockito" % "mockito-all" % "1.10.17"
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.3"
libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.5"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.1"
libraryDependencies += "com.typesafe.akka" %% "akka-stream-experimental"
% "1.0-M2"
libraryDependencies += "com.typesafe.akka" %% "akka-http-experimental"
% "1.0-M2"
libraryDependencies += "com.typesafe.akka" %% "akka-http-core-experimental"
% "1.0-M2"
libraryDependencies += "com.typesafe" % "config" % "1.2.1"
dependencyOverrides += "com.typesafe" % "config" % "1.2.1"
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with akka-http/akka-stream
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Missed the $
export
SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar:
*$SPARK_CLASSPATH*
Thanks
Best Regards
On Fri, Jan 2, 2015 at 4:57 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Can you try:
>
> export
> SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar:SPARK_CLASSPATH
>
> Also put the jar overriding the spark's version at the end.
>
> Thanks
> Best Regards
>
> On Fri, Jan 2, 2015 at 4:46 PM, Christophe Billiard <
> christophe.billiard@gmail.com> wrote:
>
>> Thank you Akhil for your idea.
>>
>> In spark-env.sh, I set
>> export
>> SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar
>>
>> When I run bin/compute-classpath.sh
>> I get Spark's classpath:
>>
>> /home/christophe/Development/spark-streaming3/config-1.2.1.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
>>
>> Few errors later, Spark's classpath looks like:
>>
>> /home/christophe/Development/spark-streaming3/config-1.2.1.jar:akka-stream-experimental_2.10-1.0-M2.jar:reactive-streams-1.0.0.M3.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
>>
>> And the error is now:
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> akka.actor.ExtendedActorSystem.systemActorOf(Lakka/actor/Props;Ljava/lang/String;)Lakka/actor/ActorRef;
>> at akka.stream.scaladsl.StreamTcp.<init>(StreamTcp.scala:147)
>> at
>> akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:140)
>> at akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:32)
>> at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:654)
>> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
>> at akka.stream.scaladsl.StreamTcp$.apply(StreamTcp.scala:134)
>> at akka.http.HttpExt.bind(Http.scala:33)
>> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:250)
>> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> I am trying to add akka-actor_2.10-2.3.7.jar to Spark's classpath:
>> And the error gets worse (Spark can't even start anymore):
>> [ERROR] [01/02/2015 12:08:14.807]
>> [sparkDriver-akka.actor.default-dispatcher-4] [ActorSystem(sparkDriver)]
>> Uncaught fatal error from thread
>> [sparkDriver-akka.actor.default-dispatcher-4] shutting down ActorSystem
>> [sparkDriver]
>> java.lang.AbstractMethodError
>> at akka.actor.ActorCell.create(ActorCell.scala:580)
>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
>> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
>> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>> at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> [ERROR] [01/02/2015 12:08:14.811]
>> [sparkDriver-akka.actor.default-dispatcher-3] [ActorSystem(sparkDriver)]
>> Uncaught fatal error from thread
>> [sparkDriver-akka.actor.default-dispatcher-3] shutting down ActorSystem
>> [sparkDriver]
>> java.lang.AbstractMethodError:
>> akka.event.slf4j.Slf4jLogger.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V
>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>> at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>> at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> My guess is that akka-actor_2.10-2.3.7.jar is overiding the akka-actor's
>> version of Spark.
>> But if I am not overidding it, I can't use akka-http/akka-stream.
>> Is there a way to work around this problem?
>>
>> Thanks,
>> Best regards
>>
>>
>> On Thu, Jan 1, 2015 at 9:28 AM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> Its a typesafe jar conflict, you will need to put the jar with
>>> getDuration method in the first position of your classpath.
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Wed, Dec 31, 2014 at 4:38 PM, Christophe Billiard <
>>> christophe.billiard@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am currently trying to combine datastax's "spark-cassandra-connector"
>>>> and
>>>> typesafe's "akka-http-experimental"
>>>> on Spark 1.1.1 (spark-cassandra-connector for Spark 1.2.0 not out yet)
>>>> and
>>>> scala 2.10.4
>>>> I am using the hadoop 2.4 pre built package. (build.sbt file at the end)
>>>>
>>>> To solve the java.lang.NoClassDefFoundError:
>>>> com/datastax/spark/connector/mapper/ColumnMapper
>>>> and other NoClassDefFoundErrors, I have to give some jars to Spark
>>>> (build.sbt is not enough).
>>>> The connectors works fine.
>>>>
>>>> My spark submit looks like:
>>>> sbt clean package; bin/spark-submit --class "SimpleAppStreaming3"
>>>> --master local[*] --jars
>>>>
>>>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar"
>>>> target/scala-2.10/simple-project_2.10-1.0.jar
>>>>
>>>> Then I am trying to add some akka-http/akka-stream features.
>>>> Like before I get a java.lang.NoClassDefFoundError:
>>>> akka/stream/FlowMaterializer$
>>>> Same solution, I begin to add jars.
>>>>
>>>> Now my spark submit looks like:
>>>> sbt clean package; bin/spark-submit --class impleAppStreaming3"
>>>> --master
>>>> local[*] --jars
>>>>
>>>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar","akka-stream-experimental_2.10-1.0-M2.jar"
>>>> target/scala-2.10/simple-project_2.10-1.0.jar
>>>>
>>>> Then I have a new kind of error:
>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>>
>>>> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>>>> at
>>>>
>>>> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>>>> at
>>>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:185)
>>>> at
>>>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:172)
>>>> at
>>>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>>>> at
>>>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>>>> at scala.Option.getOrElse(Option.scala:120)
>>>> at
>>>> akka.stream.FlowMaterializer$.apply(FlowMaterializer.scala:42)
>>>> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:240)
>>>> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
>>>> at
>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>
>>>> I can't get rid of this error.
>>>> I tried:
>>>> 1) adding several jars (including "config-1.2.1.jar")
>>>> 2) studying the dependency tree (with
>>>> https://github.com/jrudolph/sbt-dependency-graph)
>>>> 3) excluding libraryDependencies (with dependencyOverrides)
>>>>
>>>> Any ideas?
>>>>
>>>> Bonus question: Is there a way to avoid adding all these jars with
>>>> --jars?
>>>>
>>>> *My build.sbt file*
>>>>
>>>> name := "Simple Project"
>>>>
>>>> version := "1.0"
>>>>
>>>> scalaVersion := "2.10.4"
>>>>
>>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
>>>> //exclude("com.typesafe", "config")
>>>>
>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>>>>
>>>> libraryDependencies += "com.datastax.cassandra" %
>>>> "cassandra-driver-core" %
>>>> "2.1.3"
>>>>
>>>> libraryDependencies += "com.datastax.spark" %%
>>>> "spark-cassandra-connector" %
>>>> "1.1.0" withSources() withJavadoc()
>>>>
>>>> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
>>>> "2.0.5"
>>>>
>>>> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>>>>
>>>>
>>>>
>>>> libraryDependencies += "com.typesafe.akka" %% "akka-actor" %
>>>> "2.3.8"
>>>>
>>>> libraryDependencies += "com.typesafe.akka" %% "akka-testkit" %
>>>> "2.3.8"
>>>>
>>>> libraryDependencies += "org.apache.hadoop" % "hadoop-client" %
>>>> "2.4.0"
>>>>
>>>> libraryDependencies += "ch.qos.logback" % "logback-classic" %
>>>> "1.1.2"
>>>>
>>>> libraryDependencies += "org.mockito" % "mockito-all" %
>>>> "1.10.17"
>>>>
>>>> libraryDependencies += "org.scalatest" %% "scalatest" %
>>>> "2.2.3"
>>>>
>>>> libraryDependencies += "org.slf4j" % "slf4j-api" %
>>>> "1.7.5"
>>>>
>>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" %
>>>> "1.1.1"
>>>>
>>>>
>>>> libraryDependencies += "com.typesafe.akka" %% "akka-stream-experimental"
>>>> % "1.0-M2"
>>>>
>>>> libraryDependencies += "com.typesafe.akka" %% "akka-http-experimental"
>>>> % "1.0-M2"
>>>>
>>>> libraryDependencies += "com.typesafe.akka" %%
>>>> "akka-http-core-experimental"
>>>> % "1.0-M2"
>>>>
>>>>
>>>> libraryDependencies += "com.typesafe" % "config" % "1.2.1"
>>>>
>>>> dependencyOverrides += "com.typesafe" % "config" % "1.2.1"
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with akka-http/akka-stream
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Can you try:
export
SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar:SPARK_CLASSPATH
Also put the jar overriding the spark's version at the end.
Thanks
Best Regards
On Fri, Jan 2, 2015 at 4:46 PM, Christophe Billiard <
christophe.billiard@gmail.com> wrote:
> Thank you Akhil for your idea.
>
> In spark-env.sh, I set
> export
> SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar
>
> When I run bin/compute-classpath.sh
> I get Spark's classpath:
>
> /home/christophe/Development/spark-streaming3/config-1.2.1.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
>
> Few errors later, Spark's classpath looks like:
>
> /home/christophe/Development/spark-streaming3/config-1.2.1.jar:akka-stream-experimental_2.10-1.0-M2.jar:reactive-streams-1.0.0.M3.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
>
> And the error is now:
> Exception in thread "main" java.lang.NoSuchMethodError:
> akka.actor.ExtendedActorSystem.systemActorOf(Lakka/actor/Props;Ljava/lang/String;)Lakka/actor/ActorRef;
> at akka.stream.scaladsl.StreamTcp.<init>(StreamTcp.scala:147)
> at akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:140)
> at akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:32)
> at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:654)
> at akka.actor.ExtensionId$class.apply(Extension.scala:79)
> at akka.stream.scaladsl.StreamTcp$.apply(StreamTcp.scala:134)
> at akka.http.HttpExt.bind(Http.scala:33)
> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:250)
> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> I am trying to add akka-actor_2.10-2.3.7.jar to Spark's classpath:
> And the error gets worse (Spark can't even start anymore):
> [ERROR] [01/02/2015 12:08:14.807]
> [sparkDriver-akka.actor.default-dispatcher-4] [ActorSystem(sparkDriver)]
> Uncaught fatal error from thread
> [sparkDriver-akka.actor.default-dispatcher-4] shutting down ActorSystem
> [sparkDriver]
> java.lang.AbstractMethodError
> at akka.actor.ActorCell.create(ActorCell.scala:580)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> [ERROR] [01/02/2015 12:08:14.811]
> [sparkDriver-akka.actor.default-dispatcher-3] [ActorSystem(sparkDriver)]
> Uncaught fatal error from thread
> [sparkDriver-akka.actor.default-dispatcher-3] shutting down ActorSystem
> [sparkDriver]
> java.lang.AbstractMethodError:
> akka.event.slf4j.Slf4jLogger.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> My guess is that akka-actor_2.10-2.3.7.jar is overiding the akka-actor's
> version of Spark.
> But if I am not overidding it, I can't use akka-http/akka-stream.
> Is there a way to work around this problem?
>
> Thanks,
> Best regards
>
>
> On Thu, Jan 1, 2015 at 9:28 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> Its a typesafe jar conflict, you will need to put the jar with
>> getDuration method in the first position of your classpath.
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Dec 31, 2014 at 4:38 PM, Christophe Billiard <
>> christophe.billiard@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am currently trying to combine datastax's "spark-cassandra-connector"
>>> and
>>> typesafe's "akka-http-experimental"
>>> on Spark 1.1.1 (spark-cassandra-connector for Spark 1.2.0 not out yet)
>>> and
>>> scala 2.10.4
>>> I am using the hadoop 2.4 pre built package. (build.sbt file at the end)
>>>
>>> To solve the java.lang.NoClassDefFoundError:
>>> com/datastax/spark/connector/mapper/ColumnMapper
>>> and other NoClassDefFoundErrors, I have to give some jars to Spark
>>> (build.sbt is not enough).
>>> The connectors works fine.
>>>
>>> My spark submit looks like:
>>> sbt clean package; bin/spark-submit --class "SimpleAppStreaming3"
>>> --master local[*] --jars
>>>
>>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar"
>>> target/scala-2.10/simple-project_2.10-1.0.jar
>>>
>>> Then I am trying to add some akka-http/akka-stream features.
>>> Like before I get a java.lang.NoClassDefFoundError:
>>> akka/stream/FlowMaterializer$
>>> Same solution, I begin to add jars.
>>>
>>> Now my spark submit looks like:
>>> sbt clean package; bin/spark-submit --class impleAppStreaming3"
>>> --master
>>> local[*] --jars
>>>
>>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar","akka-stream-experimental_2.10-1.0-M2.jar"
>>> target/scala-2.10/simple-project_2.10-1.0.jar
>>>
>>> Then I have a new kind of error:
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>
>>> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>>> at
>>>
>>> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>>> at
>>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:185)
>>> at
>>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:172)
>>> at
>>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>>> at
>>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>>> at scala.Option.getOrElse(Option.scala:120)
>>> at akka.stream.FlowMaterializer$.apply(FlowMaterializer.scala:42)
>>> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:240)
>>> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>> I can't get rid of this error.
>>> I tried:
>>> 1) adding several jars (including "config-1.2.1.jar")
>>> 2) studying the dependency tree (with
>>> https://github.com/jrudolph/sbt-dependency-graph)
>>> 3) excluding libraryDependencies (with dependencyOverrides)
>>>
>>> Any ideas?
>>>
>>> Bonus question: Is there a way to avoid adding all these jars with
>>> --jars?
>>>
>>> *My build.sbt file*
>>>
>>> name := "Simple Project"
>>>
>>> version := "1.0"
>>>
>>> scalaVersion := "2.10.4"
>>>
>>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
>>> //exclude("com.typesafe", "config")
>>>
>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>>>
>>> libraryDependencies += "com.datastax.cassandra" %
>>> "cassandra-driver-core" %
>>> "2.1.3"
>>>
>>> libraryDependencies += "com.datastax.spark" %%
>>> "spark-cassandra-connector" %
>>> "1.1.0" withSources() withJavadoc()
>>>
>>> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
>>> "2.0.5"
>>>
>>> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>>>
>>>
>>>
>>> libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.8"
>>>
>>> libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.3.8"
>>>
>>> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.4.0"
>>>
>>> libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.2"
>>>
>>> libraryDependencies += "org.mockito" % "mockito-all" %
>>> "1.10.17"
>>>
>>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.3"
>>>
>>> libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.5"
>>>
>>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.1"
>>>
>>>
>>> libraryDependencies += "com.typesafe.akka" %% "akka-stream-experimental"
>>> % "1.0-M2"
>>>
>>> libraryDependencies += "com.typesafe.akka" %% "akka-http-experimental"
>>> % "1.0-M2"
>>>
>>> libraryDependencies += "com.typesafe.akka" %%
>>> "akka-http-core-experimental"
>>> % "1.0-M2"
>>>
>>>
>>> libraryDependencies += "com.typesafe" % "config" % "1.2.1"
>>>
>>> dependencyOverrides += "com.typesafe" % "config" % "1.2.1"
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with akka-http/akka-stream
Posted by Christophe Billiard <ch...@gmail.com>.
Thank you Akhil for your idea.
In spark-env.sh, I set
export
SPARK_CLASSPATH=/home/christophe/Development/spark-streaming3/config-1.2.1.jar
When I run bin/compute-classpath.sh
I get Spark's classpath:
/home/christophe/Development/spark-streaming3/config-1.2.1.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
Few errors later, Spark's classpath looks like:
/home/christophe/Development/spark-streaming3/config-1.2.1.jar:akka-stream-experimental_2.10-1.0-M2.jar:reactive-streams-1.0.0.M3.jar::/home/christophe/Development/spark-streaming3/conf:/home/christophe/Development/spark-streaming3/lib/spark-assembly-1.1.1-hadoop2.4.0.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-api-jdo-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-rdbms-3.2.1.jar:/home/christophe/Development/spark-streaming3/lib/datanucleus-core-3.2.2.jar
And the error is now:
Exception in thread "main" java.lang.NoSuchMethodError:
akka.actor.ExtendedActorSystem.systemActorOf(Lakka/actor/Props;Ljava/lang/String;)Lakka/actor/ActorRef;
at akka.stream.scaladsl.StreamTcp.<init>(StreamTcp.scala:147)
at akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:140)
at akka.stream.scaladsl.StreamTcp$.createExtension(StreamTcp.scala:32)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:654)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at akka.stream.scaladsl.StreamTcp$.apply(StreamTcp.scala:134)
at akka.http.HttpExt.bind(Http.scala:33)
at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:250)
at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I am trying to add akka-actor_2.10-2.3.7.jar to Spark's classpath:
And the error gets worse (Spark can't even start anymore):
[ERROR] [01/02/2015 12:08:14.807]
[sparkDriver-akka.actor.default-dispatcher-4] [ActorSystem(sparkDriver)]
Uncaught fatal error from thread
[sparkDriver-akka.actor.default-dispatcher-4] shutting down ActorSystem
[sparkDriver]
java.lang.AbstractMethodError
at akka.actor.ActorCell.create(ActorCell.scala:580)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [01/02/2015 12:08:14.811]
[sparkDriver-akka.actor.default-dispatcher-3] [ActorSystem(sparkDriver)]
Uncaught fatal error from thread
[sparkDriver-akka.actor.default-dispatcher-3] shutting down ActorSystem
[sparkDriver]
java.lang.AbstractMethodError:
akka.event.slf4j.Slf4jLogger.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
My guess is that akka-actor_2.10-2.3.7.jar is overiding the akka-actor's
version of Spark.
But if I am not overidding it, I can't use akka-http/akka-stream.
Is there a way to work around this problem?
Thanks,
Best regards
On Thu, Jan 1, 2015 at 9:28 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Its a typesafe jar conflict, you will need to put the jar with getDuration
> method in the first position of your classpath.
>
> Thanks
> Best Regards
>
> On Wed, Dec 31, 2014 at 4:38 PM, Christophe Billiard <
> christophe.billiard@gmail.com> wrote:
>
>> Hi all,
>>
>> I am currently trying to combine datastax's "spark-cassandra-connector"
>> and
>> typesafe's "akka-http-experimental"
>> on Spark 1.1.1 (spark-cassandra-connector for Spark 1.2.0 not out yet) and
>> scala 2.10.4
>> I am using the hadoop 2.4 pre built package. (build.sbt file at the end)
>>
>> To solve the java.lang.NoClassDefFoundError:
>> com/datastax/spark/connector/mapper/ColumnMapper
>> and other NoClassDefFoundErrors, I have to give some jars to Spark
>> (build.sbt is not enough).
>> The connectors works fine.
>>
>> My spark submit looks like:
>> sbt clean package; bin/spark-submit --class "SimpleAppStreaming3"
>> --master local[*] --jars
>>
>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar"
>> target/scala-2.10/simple-project_2.10-1.0.jar
>>
>> Then I am trying to add some akka-http/akka-stream features.
>> Like before I get a java.lang.NoClassDefFoundError:
>> akka/stream/FlowMaterializer$
>> Same solution, I begin to add jars.
>>
>> Now my spark submit looks like:
>> sbt clean package; bin/spark-submit --class impleAppStreaming3"
>> --master
>> local[*] --jars
>>
>> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar","akka-stream-experimental_2.10-1.0-M2.jar"
>> target/scala-2.10/simple-project_2.10-1.0.jar
>>
>> Then I have a new kind of error:
>> Exception in thread "main" java.lang.NoSuchMethodError:
>>
>> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>> at
>>
>> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>> at
>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:185)
>> at
>> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:172)
>> at
>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>> at
>> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
>> at scala.Option.getOrElse(Option.scala:120)
>> at akka.stream.FlowMaterializer$.apply(FlowMaterializer.scala:42)
>> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:240)
>> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> I can't get rid of this error.
>> I tried:
>> 1) adding several jars (including "config-1.2.1.jar")
>> 2) studying the dependency tree (with
>> https://github.com/jrudolph/sbt-dependency-graph)
>> 3) excluding libraryDependencies (with dependencyOverrides)
>>
>> Any ideas?
>>
>> Bonus question: Is there a way to avoid adding all these jars with --jars?
>>
>> *My build.sbt file*
>>
>> name := "Simple Project"
>>
>> version := "1.0"
>>
>> scalaVersion := "2.10.4"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
>> //exclude("com.typesafe", "config")
>>
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>>
>> libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core"
>> %
>> "2.1.3"
>>
>> libraryDependencies += "com.datastax.spark" %%
>> "spark-cassandra-connector" %
>> "1.1.0" withSources() withJavadoc()
>>
>> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
>> "2.0.5"
>>
>> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>>
>>
>>
>> libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.8"
>>
>> libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.3.8"
>>
>> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.4.0"
>>
>> libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.2"
>>
>> libraryDependencies += "org.mockito" % "mockito-all" %
>> "1.10.17"
>>
>> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.3"
>>
>> libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.5"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.1"
>>
>>
>> libraryDependencies += "com.typesafe.akka" %% "akka-stream-experimental"
>> % "1.0-M2"
>>
>> libraryDependencies += "com.typesafe.akka" %% "akka-http-experimental"
>> % "1.0-M2"
>>
>> libraryDependencies += "com.typesafe.akka" %%
>> "akka-http-core-experimental"
>> % "1.0-M2"
>>
>>
>> libraryDependencies += "com.typesafe" % "config" % "1.2.1"
>>
>> dependencyOverrides += "com.typesafe" % "config" % "1.2.1"
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with akka-http/akka-stream
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Its a typesafe jar conflict, you will need to put the jar with getDuration
method in the first position of your classpath.
Thanks
Best Regards
On Wed, Dec 31, 2014 at 4:38 PM, Christophe Billiard <
christophe.billiard@gmail.com> wrote:
> Hi all,
>
> I am currently trying to combine datastax's "spark-cassandra-connector" and
> typesafe's "akka-http-experimental"
> on Spark 1.1.1 (spark-cassandra-connector for Spark 1.2.0 not out yet) and
> scala 2.10.4
> I am using the hadoop 2.4 pre built package. (build.sbt file at the end)
>
> To solve the java.lang.NoClassDefFoundError:
> com/datastax/spark/connector/mapper/ColumnMapper
> and other NoClassDefFoundErrors, I have to give some jars to Spark
> (build.sbt is not enough).
> The connectors works fine.
>
> My spark submit looks like:
> sbt clean package; bin/spark-submit --class "SimpleAppStreaming3"
> --master local[*] --jars
>
> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar"
> target/scala-2.10/simple-project_2.10-1.0.jar
>
> Then I am trying to add some akka-http/akka-stream features.
> Like before I get a java.lang.NoClassDefFoundError:
> akka/stream/FlowMaterializer$
> Same solution, I begin to add jars.
>
> Now my spark submit looks like:
> sbt clean package; bin/spark-submit --class impleAppStreaming3"
> --master
> local[*] --jars
>
> "spark-cassandra-connector_2.10-1.1.0.jar","cassandra-driver-core-2.1.3.jar","cassandra-thrift-2.0.5.jar","joda-time-2.6.jar","akka-stream-experimental_2.10-1.0-M2.jar"
> target/scala-2.10/simple-project_2.10-1.0.jar
>
> Then I have a new kind of error:
> Exception in thread "main" java.lang.NoSuchMethodError:
>
> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
> at
>
> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
> at
> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:185)
> at
> akka.stream.MaterializerSettings$.apply(FlowMaterializer.scala:172)
> at
> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
> at
> akka.stream.FlowMaterializer$$anonfun$1.apply(FlowMaterializer.scala:42)
> at scala.Option.getOrElse(Option.scala:120)
> at akka.stream.FlowMaterializer$.apply(FlowMaterializer.scala:42)
> at SimpleAppStreaming3$.main(SimpleAppStreaming3.scala:240)
> at SimpleAppStreaming3.main(SimpleAppStreaming3.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> I can't get rid of this error.
> I tried:
> 1) adding several jars (including "config-1.2.1.jar")
> 2) studying the dependency tree (with
> https://github.com/jrudolph/sbt-dependency-graph)
> 3) excluding libraryDependencies (with dependencyOverrides)
>
> Any ideas?
>
> Bonus question: Is there a way to avoid adding all these jars with --jars?
>
> *My build.sbt file*
>
> name := "Simple Project"
>
> version := "1.0"
>
> scalaVersion := "2.10.4"
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
> //exclude("com.typesafe", "config")
>
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>
> libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" %
> "2.1.3"
>
> libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector"
> %
> "1.1.0" withSources() withJavadoc()
>
> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
> "2.0.5"
>
> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>
>
>
> libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.8"
>
> libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.3.8"
>
> libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.4.0"
>
> libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.2"
>
> libraryDependencies += "org.mockito" % "mockito-all" % "1.10.17"
>
> libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.3"
>
> libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.5"
>
> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.1"
>
>
> libraryDependencies += "com.typesafe.akka" %% "akka-stream-experimental"
> % "1.0-M2"
>
> libraryDependencies += "com.typesafe.akka" %% "akka-http-experimental"
> % "1.0-M2"
>
> libraryDependencies += "com.typesafe.akka" %% "akka-http-core-experimental"
> % "1.0-M2"
>
>
> libraryDependencies += "com.typesafe" % "config" % "1.2.1"
>
> dependencyOverrides += "com.typesafe" % "config" % "1.2.1"
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with
akka-http/akka-stream
Posted by Pankaj Narang <pa...@gmail.com>.
Good luck. Let me know If I can assist you further
Regards
-Pankaj
Linkedin
https://www.linkedin.com/profile/view?id=171566646
Skype
pankaj.narang
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20991.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with
akka-http/akka-stream
Posted by Christophe Billiard <ch...@gmail.com>.
Thanks Pankaj for the assembly plugin tip.
Yes there is a version mismatch of akka actor between Spark 1.1.1 and
akka-http/akka-stream (2.2.3 versus 2.3.x).
After some digging, I see 4 options for this problem (in case others
encounter it):
1) Upgrade to Spark 1.2.0, the same code will work (not possible for me
since I want to use Datastax connector)
2) Make a custom build of Spark 1.1.1
3) Use play instead with akka actor 2.2.3 (play 2.2.3 for instance)
4) Wait for Datastax connector 1.2.0 (released on 31th January 2015)
I currently trying option 3
Thank you all for your help
On Sat, Jan 3, 2015 at 4:11 AM, Pankaj Narang [via Apache Spark User List] <
ml-node+s1001560n20950h72@n3.nabble.com> wrote:
> Like before I get a java.lang.NoClassDefFoundError:
> akka/stream/FlowMaterializer$
>
> This can be solved using assembly plugin. you need to enable assembly
> plugin in global plugins
>
> C:\Users\infoshore\.sbt\0.13\plugins
> add a line in plugins.sbt addSbtPlugin("com.eed3si9n" % "sbt-assembly" %
> "0.11.0")
>
>
>
> and then add the following lines in build.sbt
>
> import AssemblyKeys._ // put this at the top of the file
>
> seq(assemblySettings: _*)
>
> Also in the bottom dont forget to add
>
> assemblySettings
>
> mergeStrategy in assembly := {
> case m if m.toLowerCase.endsWith("manifest.mf") =>
> MergeStrategy.discard
> case m if m.toLowerCase.matches("meta-inf.*\\.sf$") =>
> MergeStrategy.discard
> case "log4j.properties" =>
> MergeStrategy.discard
> case m if m.toLowerCase.startsWith("meta-inf/services/") =>
> MergeStrategy.filterDistinctLines
> case "reference.conf" =>
> MergeStrategy.concat
> case _ =>
> MergeStrategy.first
> }
>
>
> Now in your sbt run sbt assembly that will create the jar which can be run
> without --jars options
> as this will be a uber jar containing all jars
>
>
>
> Also nosuchmethod exception is thrown when there is difference in versions
> of complied and runtime versions.
>
> What is the version of spark you are using ? You need to use same version
> in build.sbt
>
>
> Here is your build.sbt
>
>
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
> //exclude("com.typesafe", "config")
>
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
>
> libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core"
> % "2.1.3"
>
> libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector"
> % "1.1.0" withSources() withJavadoc()
>
> libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" %
> "2.0.5"
>
> libraryDependencies += "joda-time" % "joda-time" % "2.6"
>
>
> and your error is Exception in thread "main" java.lang.NoSuchMethodError:
> com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
>
> at
> akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
>
> I think there is version mismatch on the jars you use at runtime
>
>
> If you need more help add me on skype pankaj.narang
>
>
> ---Pankaj
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20950.html
> To unsubscribe from NoSuchMethodError:
> com.typesafe.config.Config.getDuration with akka-http/akka-stream, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=20926&code=Y2hyaXN0b3BoZS5iaWxsaWFyZEBnbWFpbC5jb218MjA5MjZ8LTE2ODA2NTAwMDk=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20988.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: NoSuchMethodError: com.typesafe.config.Config.getDuration with
akka-http/akka-stream
Posted by Pankaj Narang <pa...@gmail.com>.
Like before I get a java.lang.NoClassDefFoundError:
akka/stream/FlowMaterializer$
This can be solved using assembly plugin. you need to enable assembly plugin
in global plugins
C:\Users\infoshore\.sbt\0.13\plugins
add a line in plugins.sbt addSbtPlugin("com.eed3si9n" % "sbt-assembly" %
"0.11.0")
and then add the following lines in build.sbt
import AssemblyKeys._ // put this at the top of the file
seq(assemblySettings: _*)
Also in the bottom dont forget to add
assemblySettings
mergeStrategy in assembly := {
case m if m.toLowerCase.endsWith("manifest.mf") =>
MergeStrategy.discard
case m if m.toLowerCase.matches("meta-inf.*\\.sf$") =>
MergeStrategy.discard
case "log4j.properties" =>
MergeStrategy.discard
case m if m.toLowerCase.startsWith("meta-inf/services/") =>
MergeStrategy.filterDistinctLines
case "reference.conf" =>
MergeStrategy.concat
case _ =>
MergeStrategy.first
}
Now in your sbt run sbt assembly that will create the jar which can be run
without --jars options
as this will be a uber jar containing all jars
Also nosuchmethod exception is thrown when there is difference in versions
of complied and runtime versions.
What is the version of spark you are using ? You need to use same version in
build.sbt
Here is your build.sbt
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.1"
//exclude("com.typesafe", "config")
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.1.1"
libraryDependencies += "com.datastax.cassandra" % "cassandra-driver-core" %
"2.1.3"
libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" %
"1.1.0" withSources() withJavadoc()
libraryDependencies += "org.apache.cassandra" % "cassandra-thrift" % "2.0.5"
libraryDependencies += "joda-time" % "joda-time" % "2.6"
and your error is Exception in thread "main" java.lang.NoSuchMethodError:
com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
at
akka.stream.StreamSubscriptionTimeoutSettings$.apply(FlowMaterializer.scala:256)
I think there is version mismatch on the jars you use at runtime
If you need more help add me on skype pankaj.narang
---Pankaj
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-com-typesafe-config-Config-getDuration-with-akka-http-akka-stream-tp20926p20950.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org