You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arthur Chan <ar...@gmail.com> on 2015/07/14 12:32:56 UTC
java.lang.IllegalStateException: unread block data
Hi,
I use Spark 1.4. When saving the model to HDFS, I got error?
Please help!
Regards
my scala command:
sc.makeRDD(model.clusterCenters,10).saveAsObjectFile("/tmp/tweets/model")
The error log:
15/07/14 18:27:40 INFO SequenceFileRDDFunctions: Saving as sequence file of
type (NullWritable,BytesWritable)
15/07/14 18:27:40 INFO SparkContext: Starting job: saveAsObjectFile at
<console>:45
15/07/14 18:27:40 INFO DAGScheduler: Got job 110 (saveAsObjectFile at
<console>:45) with 10 output partitions (allowLocal=false)
15/07/14 18:27:40 INFO DAGScheduler: Final stage: ResultStage
174(saveAsObjectFile at <console>:45)
15/07/14 18:27:40 INFO DAGScheduler: Parents of final stage: List()
15/07/14 18:27:40 INFO DAGScheduler: Missing parents: List()
15/07/14 18:27:40 INFO DAGScheduler: Submitting ResultStage 174
(MapPartitionsRDD[258] at saveAsObjectFile at <console>:45), which has no
missing parents
15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(135360) called with
curMem=14724380, maxMem=280248975
15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256 stored as values in
memory (estimated size 132.2 KB, free 253.1 MB)
15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(46231) called with
curMem=14859740, maxMem=280248975
15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256_piece0 stored as
bytes in memory (estimated size 45.1 KB, free 253.1 MB)
15/07/14 18:27:40 INFO BlockManagerInfo: Added broadcast_256_piece0 in
memory on localhost:52681 (size: 45.1 KB, free: 263.1 MB)
15/07/14 18:27:40 INFO SparkContext: Created broadcast 256 from broadcast
at DAGScheduler.scala:874
15/07/14 18:27:40 INFO DAGScheduler: Submitting 10 missing tasks from
ResultStage 174 (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45)
15/07/14 18:27:40 INFO TaskSchedulerImpl: Adding task set 174.0 with 10
tasks
15/07/14 18:27:40 INFO TaskSetManager: Starting task 0.0 in stage 174.0
(TID 4513, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 1.0 in stage 174.0
(TID 4514, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 2.0 in stage 174.0
(TID 4515, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 3.0 in stage 174.0
(TID 4516, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 4.0 in stage 174.0
(TID 4517, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 5.0 in stage 174.0
(TID 4518, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 6.0 in stage 174.0
(TID 4519, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 7.0 in stage 174.0
(TID 4520, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 8.0 in stage 174.0
(TID 4521, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 9.0 in stage 174.0
(TID 4522, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO Executor: Running task 0.0 in stage 174.0 (TID 4513)
15/07/14 18:27:40 INFO Executor: Running task 1.0 in stage 174.0 (TID 4514)
15/07/14 18:27:40 INFO Executor: Running task 2.0 in stage 174.0 (TID 4515)
15/07/14 18:27:40 INFO Executor: Running task 3.0 in stage 174.0 (TID 4516)
15/07/14 18:27:40 INFO Executor: Running task 4.0 in stage 174.0 (TID 4517)
15/07/14 18:27:40 INFO Executor: Running task 5.0 in stage 174.0 (TID 4518)
15/07/14 18:27:40 INFO Executor: Running task 6.0 in stage 174.0 (TID 4519)
15/07/14 18:27:40 INFO Executor: Running task 7.0 in stage 174.0 (TID 4520)
15/07/14 18:27:40 INFO Executor: Running task 8.0 in stage 174.0 (TID 4521)
15/07/14 18:27:40 ERROR Executor: Exception in task 1.0 in stage 174.0 (TID
4514)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR Executor: Exception in task 3.0 in stage 174.0 (TID
4516)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR Executor: Exception in task 5.0 in stage 174.0 (TID
4518)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 INFO Executor: Running task 9.0 in stage 174.0 (TID 4522)
15/07/14 18:27:40 WARN TaskSetManager: Lost task 1.0 in stage 174.0 (TID
4514, localhost): java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR TaskSetManager: Task 1 in stage 174.0 failed 1
times; aborting job
15/07/14 18:27:40 INFO TaskSetManager: Lost task 3.0 in stage 174.0 (TID
4516) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 1]
15/07/14 18:27:40 INFO TaskSchedulerImpl: Cancelling stage 174
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 6.0 in
stage 174.0 (TID 4519)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 0.0 in
stage 174.0 (TID 4513)
15/07/14 18:27:40 INFO TaskSchedulerImpl: Stage 174 was cancelled
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 7.0 in
stage 174.0 (TID 4520)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 4.0 in
stage 174.0 (TID 4517)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 8.0 in
stage 174.0 (TID 4521)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 9.0 in
stage 174.0 (TID 4522)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 2.0 in
stage 174.0 (TID 4515)
15/07/14 18:27:40 INFO DAGScheduler: ResultStage 174 (saveAsObjectFile at
<console>:45) failed in 0.005 s
15/07/14 18:27:40 INFO DAGScheduler: Job 110 failed: saveAsObjectFile at
<console>:45, took 0.016422 s
15/07/14 18:27:40 ERROR Executor: Exception in task 8.0 in stage 174.0 (TID
4521)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 INFO TaskSetManager: Lost task 5.0 in stage 174.0 (TID
4518) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 2]
15/07/14 18:27:40 INFO TaskSetManager: Lost task 8.0 in stage 174.0 (TID
4521) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 3]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 174.0 failed 1 times, most recent failure: Lost task 1.0 in stage
174.0 (TID 4514, localhost): java.lang.IllegalStateException: unread block
data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Re: java.lang.IllegalStateException: unread block data
Posted by Arthur Chan <ar...@gmail.com>.
I found the reason, it is about sc. Thanks
On Tue, Jul 14, 2015 at 9:45 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Someone else also reported this error with spark 1.4.0
>
> Thanks
> Best Regards
>
> On Tue, Jul 14, 2015 at 6:57 PM, Arthur Chan <ar...@gmail.com>
> wrote:
>
>> Hi, Below is the log form the worker.
>>
>>
>> 15/07/14 17:18:56 ERROR FileAppender: Error writing stream to file
>> /spark/app-20150714171703-0004/5/stderr
>>
>> java.io.IOException: Stream closed
>>
>> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
>>
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
>>
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>
>> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>>
>> at
>> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>>
>> at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>>
>> at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>>
>> at
>> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>>
>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
>>
>> at
>> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
>>
>> 15/07/14 17:18:57 INFO Worker: Executor app-20150714171703-0004/5
>> finished with state KILLED exitStatus 143
>>
>> 15/07/14 17:18:57 INFO Worker: Cleaning up local directories for
>> application app-20150714171703-0004
>>
>> 15/07/14 17:18:57 WARN ReliableDeliverySupervisor: Association with
>> remote system [akka.tcp://sparkExecutor@10.10.10.1:52635] has failed,
>> address is now gated for [5000] ms. Reason is: [Disassociated].
>>
>
>
Re: java.lang.IllegalStateException: unread block data
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Someone else also reported this error with spark 1.4.0
Thanks
Best Regards
On Tue, Jul 14, 2015 at 6:57 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi, Below is the log form the worker.
>
>
> 15/07/14 17:18:56 ERROR FileAppender: Error writing stream to file
> /spark/app-20150714171703-0004/5/stderr
>
> java.io.IOException: Stream closed
>
> at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
>
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:107)
>
> at
> org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
>
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
>
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>
> at
> org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
>
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
>
> at
> org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
>
> 15/07/14 17:18:57 INFO Worker: Executor app-20150714171703-0004/5 finished
> with state KILLED exitStatus 143
>
> 15/07/14 17:18:57 INFO Worker: Cleaning up local directories for
> application app-20150714171703-0004
>
> 15/07/14 17:18:57 WARN ReliableDeliverySupervisor: Association with remote
> system [akka.tcp://sparkExecutor@10.10.10.1:52635] has failed, address is
> now gated for [5000] ms. Reason is: [Disassociated].
>
Re: java.lang.IllegalStateException: unread block data
Posted by Arthur Chan <ar...@gmail.com>.
Hi, Below is the log form the worker.
15/07/14 17:18:56 ERROR FileAppender: Error writing stream to file
/spark/app-20150714171703-0004/5/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at
org.apache.spark.util.logging.FileAppender.appendStreamToFile(FileAppender.scala:70)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply$mcV$sp(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at
org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$run$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772)
at
org.apache.spark.util.logging.FileAppender$$anon$1.run(FileAppender.scala:38)
15/07/14 17:18:57 INFO Worker: Executor app-20150714171703-0004/5 finished
with state KILLED exitStatus 143
15/07/14 17:18:57 INFO Worker: Cleaning up local directories for
application app-20150714171703-0004
15/07/14 17:18:57 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@10.10.10.1:52635] has failed, address is
now gated for [5000] ms. Reason is: [Disassociated].
Re: java.lang.IllegalStateException: unread block data
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Look in the worker logs and see whats going on.
Thanks
Best Regards
On Tue, Jul 14, 2015 at 4:02 PM, Arthur Chan <ar...@gmail.com>
wrote:
> Hi,
>
> I use Spark 1.4. When saving the model to HDFS, I got error?
>
> Please help!
> Regards
>
>
>
> my scala command:
> sc.makeRDD(model.clusterCenters,10).saveAsObjectFile("/tmp/tweets/model")
>
> The error log:
>
> 15/07/14 18:27:40 INFO SequenceFileRDDFunctions: Saving as sequence file
> of type (NullWritable,BytesWritable)
>
> 15/07/14 18:27:40 INFO SparkContext: Starting job: saveAsObjectFile at
> <console>:45
>
> 15/07/14 18:27:40 INFO DAGScheduler: Got job 110 (saveAsObjectFile at
> <console>:45) with 10 output partitions (allowLocal=false)
>
> 15/07/14 18:27:40 INFO DAGScheduler: Final stage: ResultStage
> 174(saveAsObjectFile at <console>:45)
>
> 15/07/14 18:27:40 INFO DAGScheduler: Parents of final stage: List()
>
> 15/07/14 18:27:40 INFO DAGScheduler: Missing parents: List()
>
> 15/07/14 18:27:40 INFO DAGScheduler: Submitting ResultStage 174
> (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45), which has no
> missing parents
>
> 15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(135360) called with
> curMem=14724380, maxMem=280248975
>
> 15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256 stored as values
> in memory (estimated size 132.2 KB, free 253.1 MB)
>
> 15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(46231) called with
> curMem=14859740, maxMem=280248975
>
> 15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256_piece0 stored as
> bytes in memory (estimated size 45.1 KB, free 253.1 MB)
>
> 15/07/14 18:27:40 INFO BlockManagerInfo: Added broadcast_256_piece0 in
> memory on localhost:52681 (size: 45.1 KB, free: 263.1 MB)
>
> 15/07/14 18:27:40 INFO SparkContext: Created broadcast 256 from broadcast
> at DAGScheduler.scala:874
>
> 15/07/14 18:27:40 INFO DAGScheduler: Submitting 10 missing tasks from
> ResultStage 174 (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45)
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Adding task set 174.0 with 10
> tasks
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 0.0 in stage 174.0
> (TID 4513, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 1.0 in stage 174.0
> (TID 4514, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 2.0 in stage 174.0
> (TID 4515, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 3.0 in stage 174.0
> (TID 4516, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 4.0 in stage 174.0
> (TID 4517, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 5.0 in stage 174.0
> (TID 4518, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 6.0 in stage 174.0
> (TID 4519, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 7.0 in stage 174.0
> (TID 4520, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 8.0 in stage 174.0
> (TID 4521, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Starting task 9.0 in stage 174.0
> (TID 4522, localhost, PROCESS_LOCAL, 9486 bytes)
>
> 15/07/14 18:27:40 INFO Executor: Running task 0.0 in stage 174.0 (TID 4513)
>
> 15/07/14 18:27:40 INFO Executor: Running task 1.0 in stage 174.0 (TID 4514)
>
> 15/07/14 18:27:40 INFO Executor: Running task 2.0 in stage 174.0 (TID 4515)
>
> 15/07/14 18:27:40 INFO Executor: Running task 3.0 in stage 174.0 (TID 4516)
>
> 15/07/14 18:27:40 INFO Executor: Running task 4.0 in stage 174.0 (TID 4517)
>
> 15/07/14 18:27:40 INFO Executor: Running task 5.0 in stage 174.0 (TID 4518)
>
> 15/07/14 18:27:40 INFO Executor: Running task 6.0 in stage 174.0 (TID 4519)
>
> 15/07/14 18:27:40 INFO Executor: Running task 7.0 in stage 174.0 (TID 4520)
>
> 15/07/14 18:27:40 INFO Executor: Running task 8.0 in stage 174.0 (TID 4521)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 1.0 in stage 174.0
> (TID 4514)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 3.0 in stage 174.0
> (TID 4516)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 5.0 in stage 174.0
> (TID 4518)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 INFO Executor: Running task 9.0 in stage 174.0 (TID 4522)
>
> 15/07/14 18:27:40 WARN TaskSetManager: Lost task 1.0 in stage 174.0 (TID
> 4514, localhost): java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> 15/07/14 18:27:40 ERROR TaskSetManager: Task 1 in stage 174.0 failed 1
> times; aborting job
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 3.0 in stage 174.0 (TID
> 4516) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 1]
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Cancelling stage 174
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 6.0 in
> stage 174.0 (TID 4519)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 0.0 in
> stage 174.0 (TID 4513)
>
> 15/07/14 18:27:40 INFO TaskSchedulerImpl: Stage 174 was cancelled
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 7.0 in
> stage 174.0 (TID 4520)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 4.0 in
> stage 174.0 (TID 4517)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 8.0 in
> stage 174.0 (TID 4521)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 9.0 in
> stage 174.0 (TID 4522)
>
> 15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 2.0 in
> stage 174.0 (TID 4515)
>
> 15/07/14 18:27:40 INFO DAGScheduler: ResultStage 174 (saveAsObjectFile at
> <console>:45) failed in 0.005 s
>
> 15/07/14 18:27:40 INFO DAGScheduler: Job 110 failed: saveAsObjectFile at
> <console>:45, took 0.016422 s
>
> 15/07/14 18:27:40 ERROR Executor: Exception in task 8.0 in stage 174.0
> (TID 4521)
>
> java.lang.IllegalStateException: unread block data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 5.0 in stage 174.0 (TID
> 4518) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 2]
>
> 15/07/14 18:27:40 INFO TaskSetManager: Lost task 8.0 in stage 174.0 (TID
> 4521) on executor localhost: java.lang.IllegalStateException (unread block
> data) [duplicate 3]
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
> in stage 174.0 failed 1 times, most recent failure: Lost task 1.0 in stage
> 174.0 (TID 4514, localhost): java.lang.IllegalStateException: unread block
> data
>
> at
> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
> Driver stacktrace:
>
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256)
>
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
>
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
>
> at scala.Option.foreach(Option.scala:257)
>
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
>
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450)
>
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411)
>
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>