You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeff Jones <jj...@adaptivebiotech.com> on 2015/08/07 01:12:52 UTC
All masters are unresponsive! Giving up.
I wrote a very simple Spark 1.4.1 app that I can run through a local driver program just fine using setMaster("local[*]"). The app is as follows:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
object Hello {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]")
val sc = new SparkContext(conf)
val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
println("Max: " + data.max)
println("Min: " + data.min)
}
}
I compile this using the following build.sbt which will pull the needed Spark libraries for me.
name := """SparkyJeff"""
version := "1.0"
scalaVersion := "2.11.6"
// Change this to another test framework if you prefer
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1",
"org.apache.spark" %% "spark-sql" % "1.4.1")
// Uncomment to use Akka
//libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
fork := true
Now I'm trying to run this against a standalone cluster by changing the setMaster("local[*]") to setMaster("spark://p3.ourdomain.com:7077"). I downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP="p3.ourdomain.com", SPARK_WORKER_CORES="1000",SPARK_WORKER_MEMORY="500g" and then started the cluster using run-all.sh. The cluster appears to start fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same master URL as mentioned above.
Now when I run my little app I get the following client error:
...
[error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
...
Looking into the master logs I find:
15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877 got disassociated, removing it.
15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
java.io.InvalidClassException: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Any help would be greatly appreciated.
Thanks,
Jeff
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
Re: All masters are unresponsive! Giving up.
Posted by Igor Berman <ig...@gmail.com>.
check on which ip/port master listens
netstat -a -t --numeric-ports
On 7 August 2015 at 20:48, Jeff Jones <jj...@adaptivebiotech.com> wrote:
> Thanks. Added this to both the client and the master but still not getting
> any more information. I confirmed the flag with ps.
>
>
>
> jjones 53222 2.7 0.1 19399412 549656 pts/3 Sl 17:17 0:44
> /opt/jdk1.8/bin/java -cp
> /home/jjones/bin/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
> -Dsun.io.serialization.extendedDebugInfo=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip p3.ourdomain.com --port 7077
> --webui-port 8080’
>
>
>
> Error message(s) the same:
>
>
>
> 15/08/07 17:23:26 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payloadClass$1(Endpoint.scala:59)
>
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:99)
>
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
>
>
> *From:* Sonal Goyal [mailto:sonalgoyal4@gmail.com]
> *Sent:* Thursday, August 6, 2015 11:22 PM
> *To:* Jeff Jones
> *Cc:* user@spark.apache.org
> *Subject:* Re: All masters are unresponsive! Giving up.
>
>
>
> There seems to be a version mismatch somewhere. You can try and find out
> the cause with debug serialization information. I think the jvm flag
> -Dsun.io.*serialization*.*extendedDebugInfo*=true should help.
>
>
> Best Regards,
> Sonal
> Founder, Nube Technologies <http://www.nubetech.co>
>
> Check out Reifier at Spark Summit 2015
> <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/>
>
>
>
>
>
> On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
> I wrote a very simple Spark 1.4.1 app that I can run through a local
> driver program just fine using setMaster(“local[*]”). The app is as
> follows:
>
>
>
> import org.apache.spark.SparkContext
>
> import org.apache.spark.SparkContext._
>
> import org.apache.spark.SparkConf
>
> import org.apache.spark.rdd.RDD
>
>
>
> object Hello {
>
> def main(args: Array[String]): Unit = {
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("local[*]")
>
> val sc = new SparkContext(conf)
>
> val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
>
> println("Max: " + data.max)
>
> println("Min: " + data.min)
>
> }
>
> }
>
>
>
> I compile this using the following build.sbt which will pull the needed
> Spark libraries for me.
>
>
>
> name := """SparkyJeff"""
>
>
>
> version := "1.0"
>
>
>
> scalaVersion := "2.11.6"
>
>
>
> // Change this to another test framework if you prefer
>
> libraryDependencies ++= Seq(
>
> "org.apache.spark" %% "spark-core" % "1.4.1",
>
> "org.apache.spark" %% "spark-sql" % "1.4.1")
>
>
>
> // Uncomment to use Akka
>
> //libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
>
>
>
> fork := true
>
>
>
> Now I’m trying to run this against a standalone cluster by changing the
> setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I
> downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP=”
> p3.ourdomain.com”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g”
> and then started the cluster using run-all.sh. The cluster appears to start
> fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same
> master URL as mentioned above.
>
>
>
> Now when I run my little app I get the following client error:
>
>
>
> …
>
> [error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application
> has been killed. Reason: All masters are unresponsive! Giving up.
>
> …
>
>
>
> Looking into the master logs I find:
>
>
>
> 15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877
> got disassociated, removing it.
>
> 15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
>
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Thanks,
> Jeff
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
Re: All masters are unresponsive! Giving up.
Posted by Ted Yu <yu...@gmail.com>.
Spark 1.4.1 depends on:
<akka.version>2.3.4-spark</akka.version>
Is it possible that your standalone cluster has another version of akka ?
Cheers
On Fri, Aug 7, 2015 at 10:48 AM, Jeff Jones <jj...@adaptivebiotech.com>
wrote:
> Thanks. Added this to both the client and the master but still not getting
> any more information. I confirmed the flag with ps.
>
>
>
> jjones 53222 2.7 0.1 19399412 549656 pts/3 Sl 17:17 0:44
> /opt/jdk1.8/bin/java -cp
> /home/jjones/bin/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar
> -Dsun.io.serialization.extendedDebugInfo=true -Xms512m -Xmx512m
> org.apache.spark.deploy.master.Master --ip p3.ourdomain.com --port 7077
> --webui-port 8080’
>
>
>
> Error message(s) the same:
>
>
>
> 15/08/07 17:23:26 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payloadClass$1(Endpoint.scala:59)
>
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:99)
>
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
>
>
> *From:* Sonal Goyal [mailto:sonalgoyal4@gmail.com]
> *Sent:* Thursday, August 6, 2015 11:22 PM
> *To:* Jeff Jones
> *Cc:* user@spark.apache.org
> *Subject:* Re: All masters are unresponsive! Giving up.
>
>
>
> There seems to be a version mismatch somewhere. You can try and find out
> the cause with debug serialization information. I think the jvm flag
> -Dsun.io.*serialization*.*extendedDebugInfo*=true should help.
>
>
> Best Regards,
> Sonal
> Founder, Nube Technologies <http://www.nubetech.co>
>
> Check out Reifier at Spark Summit 2015
> <https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/>
>
>
>
>
>
> On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jj...@adaptivebiotech.com>
> wrote:
>
> I wrote a very simple Spark 1.4.1 app that I can run through a local
> driver program just fine using setMaster(“local[*]”). The app is as
> follows:
>
>
>
> import org.apache.spark.SparkContext
>
> import org.apache.spark.SparkContext._
>
> import org.apache.spark.SparkConf
>
> import org.apache.spark.rdd.RDD
>
>
>
> object Hello {
>
> def main(args: Array[String]): Unit = {
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("local[*]")
>
> val sc = new SparkContext(conf)
>
> val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
>
> println("Max: " + data.max)
>
> println("Min: " + data.min)
>
> }
>
> }
>
>
>
> I compile this using the following build.sbt which will pull the needed
> Spark libraries for me.
>
>
>
> name := """SparkyJeff"""
>
>
>
> version := "1.0"
>
>
>
> scalaVersion := "2.11.6"
>
>
>
> // Change this to another test framework if you prefer
>
> libraryDependencies ++= Seq(
>
> "org.apache.spark" %% "spark-core" % "1.4.1",
>
> "org.apache.spark" %% "spark-sql" % "1.4.1")
>
>
>
> // Uncomment to use Akka
>
> //libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
>
>
>
> fork := true
>
>
>
> Now I’m trying to run this against a standalone cluster by changing the
> setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I
> downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP=”
> p3.ourdomain.com”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g”
> and then started the cluster using run-all.sh. The cluster appears to start
> fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same
> master URL as mentioned above.
>
>
>
> Now when I run my little app I get the following client error:
>
>
>
> …
>
> [error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application
> has been killed. Reason: All masters are unresponsive! Giving up.
>
> …
>
>
>
> Looking into the master logs I find:
>
>
>
> 15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877
> got disassociated, removing it.
>
> 15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
>
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Thanks,
> Jeff
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
RE: All masters are unresponsive! Giving up.
Posted by Jeff Jones <jj...@adaptivebiotech.com>.
Thanks. Added this to both the client and the master but still not getting any more information. I confirmed the flag with ps.
jjones 53222 2.7 0.1 19399412 549656 pts/3 Sl 17:17 0:44 /opt/jdk1.8/bin/java -cp /home/jjones/bin/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/jjones/bin/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar -Dsun.io.serialization.extendedDebugInfo=true -Xms512m -Xmx512m org.apache.spark.deploy.master.Master --ip p3.ourdomain.com --port 7077 --webui-port 8080’
Error message(s) the same:
15/08/07 17:23:26 ERROR Remoting: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
java.io.InvalidClassException: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payloadClass$1(Endpoint.scala:59)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:99)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
From: Sonal Goyal [mailto:sonalgoyal4@gmail.com]
Sent: Thursday, August 6, 2015 11:22 PM
To: Jeff Jones
Cc: user@spark.apache.org
Subject: Re: All masters are unresponsive! Giving up.
There seems to be a version mismatch somewhere. You can try and find out the cause with debug serialization information. I think the jvm flag -Dsun.io.serialization.extendedDebugInfo=true should help.
Best Regards,
Sonal
Founder, Nube Technologies<http://www.nubetech.co>
Check out Reifier at Spark Summit 2015<https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/>
On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jj...@adaptivebiotech.com>> wrote:
I wrote a very simple Spark 1.4.1 app that I can run through a local driver program just fine using setMaster(“local[*]”). The app is as follows:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
object Hello {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]")
val sc = new SparkContext(conf)
val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
println("Max: " + data.max)
println("Min: " + data.min)
}
}
I compile this using the following build.sbt which will pull the needed Spark libraries for me.
name := """SparkyJeff"""
version := "1.0"
scalaVersion := "2.11.6"
// Change this to another test framework if you prefer
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1",
"org.apache.spark" %% "spark-sql" % "1.4.1")
// Uncomment to use Akka
//libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
fork := true
Now I’m trying to run this against a standalone cluster by changing the setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP=”p3.ourdomain.com<http://p3.ourdomain.com>”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g” and then started the cluster using run-all.sh. The cluster appears to start fine. I can hit cluster UI at p3.ourdomain.com:8080<http://p3.ourdomain.com:8080> and see the same master URL as mentioned above.
Now when I run my little app I get the following client error:
…
[error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master.<http://sparkMaster@p3.ourdomain.com:7077/user/Master.>..
[error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077<http://sparkMaster@p3.ourdomain.com:7077>] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master.<http://sparkMaster@p3.ourdomain.com:7077/user/Master.>..
[error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077<http://sparkMaster@p3.ourdomain.com:7077>] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master.<http://sparkMaster@p3.ourdomain.com:7077/user/Master.>..
[error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077<http://sparkMaster@p3.ourdomain.com:7077>] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
…
Looking into the master logs I find:
15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877<http://sparkDriver@192.168.137.41:48877> got disassociated, removing it.
15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
java.io.InvalidClassException: org.apache.spark.deploy.Command; local class incompatible: stream classdesc serialVersionUID = -7098307370860582211, local class serialVersionUID = -3335312719467547622
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Any help would be greatly appreciated.
Thanks,
Jeff
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
This message (and any attachments) is intended only for the designated recipient(s). It
may contain confidential or proprietary information, or have other limitations on use as
indicated by the sender. If you are not a designated recipient, you may not review, use,
copy or distribute this message. If you received this in error, please notify the sender by
reply e-mail and delete this message.
Re: All masters are unresponsive! Giving up.
Posted by Sonal Goyal <so...@gmail.com>.
There seems to be a version mismatch somewhere. You can try and find out
the cause with debug serialization information. I think the jvm flag
-Dsun.io.serialization.extendedDebugInfo=true should help.
Best Regards,
Sonal
Founder, Nube Technologies <http://www.nubetech.co>
Check out Reifier at Spark Summit 2015
<https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/>
<http://in.linkedin.com/in/sonalgoyal>
On Fri, Aug 7, 2015 at 4:42 AM, Jeff Jones <jj...@adaptivebiotech.com>
wrote:
> I wrote a very simple Spark 1.4.1 app that I can run through a local
> driver program just fine using setMaster(“local[*]”). The app is as
> follows:
>
>
>
> import org.apache.spark.SparkContext
>
> import org.apache.spark.SparkContext._
>
> import org.apache.spark.SparkConf
>
> import org.apache.spark.rdd.RDD
>
>
>
> object Hello {
>
> def main(args: Array[String]): Unit = {
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("local[*]")
>
> val sc = new SparkContext(conf)
>
> val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
>
> println("Max: " + data.max)
>
> println("Min: " + data.min)
>
> }
>
> }
>
>
>
> I compile this using the following build.sbt which will pull the needed
> Spark libraries for me.
>
>
>
> name := """SparkyJeff"""
>
>
>
> version := "1.0"
>
>
>
> scalaVersion := "2.11.6"
>
>
>
> // Change this to another test framework if you prefer
>
> libraryDependencies ++= Seq(
>
> "org.apache.spark" %% "spark-core" % "1.4.1",
>
> "org.apache.spark" %% "spark-sql" % "1.4.1")
>
>
>
> // Uncomment to use Akka
>
> //libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
>
>
>
> fork := true
>
>
>
> Now I’m trying to run this against a standalone cluster by changing the
> setMaster(“local[*]”) to setMaster(“spark://p3.ourdomain.com:7077”). I
> downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the SPARK_MASTER_IP=”
> p3.ourdomain.com”, SPARK_WORKER_CORES=”1000”,SPARK_WORKER_MEMORY=”500g”
> and then started the cluster using run-all.sh. The cluster appears to start
> fine. I can hit cluster UI at p3.ourdomain.com:8080 and see the same
> master URL as mentioned above.
>
>
>
> Now when I run my little app I get the following client error:
>
>
>
> …
>
> [error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@p3.ourdomain.com:7077/user/Master...
>
> [error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association
> with remote system [akka.tcp://sparkMaster@p3.ourdomain.com:7077] has
> failed, address is now gated for [5000] ms. Reason is: [Disassociated].
>
> [error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application
> has been killed. Reason: All masters are unresponsive! Giving up.
>
> …
>
>
>
> Looking into the master logs I find:
>
>
>
> 15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877
> got disassociated, removing it.
>
> 15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> java.io.InvalidClassException: org.apache.spark.deploy.Command; local
> class incompatible: stream classdesc serialVersionUID =
> -7098307370860582211, local class serialVersionUID = -3335312719467547622
>
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
>
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
>
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
>
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
>
> at
> akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
>
> at
> akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
>
> at scala.util.Try$.apply(Try.scala:161)
>
> at
> akka.serialization.Serialization.deserialize(Serialization.scala:98)
>
> at
> akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
>
> at
> akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
>
> at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
>
> at
> akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
>
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
> at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
>
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>
> at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Thanks,
> Jeff
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>