You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Niranda Perera <ni...@gmail.com> on 2015/01/06 05:36:01 UTC
Spark avro: Sample app fails to run in a spark standalone cluster
Hi,
I have this simple spark app.
public class AvroSparkTest {
public static void main(String[] args) throws Exception {
SparkConf sparkConf = new SparkConf()
.setMaster("spark://niranda-ThinkPad-T540p:7077") //
("local[2]")
.setAppName("avro-spark-test");
JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
JavaSQLContext sqlContext = new JavaSQLContext(sparkContext);
JavaSchemaRDD episodes = AvroUtils.avroFile(sqlContext,
"/home/niranda/projects/avro-spark-test/src/test/resources/episodes.avro");
episodes.printSchema();
episodes.registerTempTable("avroTable");
List<Row> result = sqlContext.sql("SELECT * FROM
avroTable").collect();
for (Row row : result) {
System.out.println(row.toString());
}
}
}
It works well with master being set to local.
But when I set master to a local spark server with 2 local workers, it
gives the following error.
15/01/06 10:00:55 INFO MemoryStore: ensureFreeSpace(177166) called with
curMem=0, maxMem=2004174766
15/01/06 10:00:55 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 173.0 KB, free 1911.2 MB)
15/01/06 10:00:55 INFO MemoryStore: ensureFreeSpace(25502) called with
curMem=177166, maxMem=2004174766
15/01/06 10:00:55 INFO MemoryStore: Block broadcast_0_piece0 stored as
bytes in memory (estimated size 24.9 KB, free 1911.1 MB)
15/01/06 10:00:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory
on 10.100.5.109:41873 (size: 24.9 KB, free: 1911.3 MB)
15/01/06 10:00:55 INFO BlockManagerMaster: Updated info of block
broadcast_0_piece0
15/01/06 10:00:55 INFO SparkContext: Created broadcast 0 from hadoopFile at
AvroRelation.scala:45
15/01/06 10:00:55 INFO FileInputFormat: Total input paths to process : 1
15/01/06 10:00:55 INFO SparkContext: Starting job: collect at
SparkPlan.scala:84
15/01/06 10:00:55 INFO DAGScheduler: Got job 0 (collect at
SparkPlan.scala:84) with 2 output partitions (allowLocal=false)
15/01/06 10:00:55 INFO DAGScheduler: Final stage: Stage 0(collect at
SparkPlan.scala:84)
15/01/06 10:00:55 INFO DAGScheduler: Parents of final stage: List()
15/01/06 10:00:55 INFO DAGScheduler: Missing parents: List()
15/01/06 10:00:55 INFO DAGScheduler: Submitting Stage 0 (MappedRDD[6] at
map at SparkPlan.scala:84), which has no missing parents
15/01/06 10:00:55 INFO MemoryStore: ensureFreeSpace(4864) called with
curMem=202668, maxMem=2004174766
15/01/06 10:00:55 INFO MemoryStore: Block broadcast_1 stored as values in
memory (estimated size 4.8 KB, free 1911.1 MB)
15/01/06 10:00:55 INFO MemoryStore: ensureFreeSpace(3482) called with
curMem=207532, maxMem=2004174766
15/01/06 10:00:55 INFO MemoryStore: Block broadcast_1_piece0 stored as
bytes in memory (estimated size 3.4 KB, free 1911.1 MB)
15/01/06 10:00:55 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory
on 10.100.5.109:41873 (size: 3.4 KB, free: 1911.3 MB)
15/01/06 10:00:55 INFO BlockManagerMaster: Updated info of block
broadcast_1_piece0
15/01/06 10:00:55 INFO SparkContext: Created broadcast 1 from broadcast at
DAGScheduler.scala:838
15/01/06 10:00:55 INFO DAGScheduler: Submitting 2 missing tasks from Stage
0 (MappedRDD[6] at map at SparkPlan.scala:84)
15/01/06 10:00:55 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
15/01/06 10:00:55 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID
0, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID
1, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1,
10.100.5.109): java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2722)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1009)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at
org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 1]
15/01/06 10:00:55 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
2, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID
3, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID 2)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 2]
15/01/06 10:00:55 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID
4, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 3]
15/01/06 10:00:55 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID
5, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 4]
15/01/06 10:00:55 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID
6, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 5]
15/01/06 10:00:55 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID
7, 10.100.5.109, PROCESS_LOCAL, 1340 bytes)
15/01/06 10:00:55 INFO TaskSetManager: Lost task 0.3 in stage 0.0 (TID 6)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 6]
15/01/06 10:00:55 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times;
aborting job
15/01/06 10:00:55 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 7)
on executor 10.100.5.109: java.io.EOFException (null) [duplicate 7]
15/01/06 10:00:55 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks
have all completed, from pool
15/01/06 10:00:55 INFO TaskSchedulerImpl: Cancelling stage 0
15/01/06 10:00:55 INFO DAGScheduler: Job 0 failed: collect at
SparkPlan.scala:84, took 0.127572 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due
to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 0.0 (TID 6, 10.100.5.109): java.io.EOFException
at
java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2722)
at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1009)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at
org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.UTF8.readChars(UTF8.java:216)
at org.apache.hadoop.io.UTF8.readString(UTF8.java:208)
at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
at
org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:237)
at
org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66)
at
org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:43)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:985)
at
org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1969)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1775)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1327)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
what might be the reason behind this? I'm using intelliJ idea as my IDE
I have attached the logs herewith
Rgds
--
Niranda Perera