You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by "Ajay Viswanathan (JIRA)" <ji...@apache.org> on 2014/06/24 11:02:24 UTC
[jira] [Created] (MESOS-1535) Pyspark on Mesos scheduler error
Ajay Viswanathan created MESOS-1535:
---------------------------------------
Summary: Pyspark on Mesos scheduler error
Key: MESOS-1535
URL: https://issues.apache.org/jira/browse/MESOS-1535
Project: Mesos
Issue Type: Bug
Affects Versions: 0.18.0, 0.18.1
Environment: Running a Mesos on a cluster of Centos 6.5 machines. 180 GB memory.
Reporter: Ajay Viswanathan
This is an error that I get while running fine-grained PySpark on the mesos cluster. This comes after running some 200-1000 tasks generally.
Pyspark code:
while True:
sc.parallelize(range(10)).map(lambda n : n*2).collect()
Error log:
org.apache.spark.SparkException: EOF reached before Python server acknowledged
at org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonR DD.scala:322)
at org.apache.spark.api.python.PythonAccumulatorParam.addInPlace(PythonR DD.scala:293)
at org.apache.spark.Accumulable.$plus$plus$eq(Accumulators.scala:70)
at org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala :275)
at org.apache.spark.Accumulators$$anonfun$add$2.apply(Accumulators.scala :273)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply( TraversableLike.scala:772)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca la:98)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca la:98)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala :226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.s cala:771)
at org.apache.spark.Accumulators$.add(Accumulators.scala:273)
at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGSched uler.scala:826)
at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.sca la:601)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$ano nfun$receive$1.applyOrElse(DAGScheduler.scala:190)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abst ractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool .java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:19 79)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThre ad.java:107)
--
This message was sent by Atlassian JIRA
(v6.2#6252)