You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chen Song <ch...@gmail.com> on 2014/08/11 16:29:39 UTC
Re: saveAsTextFiles file not found exception
I got the same exception after the streaming job runs for a while, The
ERROR message was complaining about a temp file not being found in the
output folder.
14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
1407744300000 ms.0
java.io.FileNotFoundException: File
hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
at
org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
at
org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
at
org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
at
org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at
org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
at
org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
wrote:
> I just saw another error after my job was run for 2 hours:
>
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>
>
>
> All my jobs use the same parameter to the function checkpoint. Is it the
> reason for the error?
>
> I will post the stack trace of the other error after it appears again.
> Thanks!
>
>
> Bill
>
>
> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
> tathagata.das1565@gmail.com> wrote:
>
>> Can you give a stack trace and logs of the exception? Its hard to say
>> anything without any associated stack trace and logs.
>>
>> TD
>>
>>
>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <bi...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am running a Spark Streaming job that uses saveAsTextFiles to save
>>> results into hdfs files. However, it has an exception after 20 batches
>>>
>>>
>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>
>>>
>>> When the job is running, I do not change any file in the folder. Does
>>> anyone know why the file cannot be found?
>>>
>>> Thanks!
>>>
>>> Bill
>>>
>>
>>
>
--
Chen Song
Re: saveAsTextFiles file not found exception
Posted by Chen Song <ch...@gmail.com>.
Thanks for putting this together, Andrew.
On Tue, Aug 12, 2014 at 2:11 AM, Andrew Ash <an...@andrewash.com> wrote:
> Hi Chen,
>
> Please see the bug I filed at
> https://issues.apache.org/jira/browse/SPARK-2984 with the
> FileNotFoundException on _temporary directory issue.
>
> Andrew
>
>
> On Mon, Aug 11, 2014 at 10:50 PM, Andrew Ash <an...@andrewash.com> wrote:
>
>> Not sure which stalled HDFS client issue your'e referring to, but there
>> was one fixed in Spark 1.0.2 that could help you out --
>> https://github.com/apache/spark/pull/1409. I've still seen one related
>> to Configuration objects not being threadsafe though so you'd still need to
>> keep speculation on to fix that (SPARK-2546)
>>
>> As it stands now, I can:
>>
>> A) have speculation off, in which case I get random hangs for a variety
>> of reasons (your HDFS stall, my Configuration safety issue)
>>
>> or
>>
>> B) have speculation on, in which case I get random failures related to
>> LeaseExpiredExceptions and .../_temporary/... file doesn't exist exceptions.
>>
>>
>> Kind of a catch-22 -- there's no reliable way to run large jobs on Spark
>> right now!
>>
>> I'm going to file a bug for the _temporary and LeaseExpiredExceptions as
>> I think these are widespread enough that we need a place to track a
>> resolution.
>>
>>
>> On Mon, Aug 11, 2014 at 9:08 AM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> Andrew that is a good finding.
>>>
>>> Yes, I have speculative execution turned on, becauseI saw tasks stalled
>>> on HDFS client.
>>>
>>> If I turned off speculative execution, is there a way to circumvent the
>>> hanging task issue?
>>>
>>>
>>>
>>> On Mon, Aug 11, 2014 at 11:13 AM, Andrew Ash <an...@andrewash.com>
>>> wrote:
>>>
>>>> I've also been seeing similar stacktraces on Spark core (not streaming)
>>>> and have a theory it's related to spark.speculation being turned on. Do
>>>> you have that enabled by chance?
>>>>
>>>>
>>>> On Mon, Aug 11, 2014 at 8:10 AM, Chen Song <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> Bill
>>>>>
>>>>> Did you get this resolved somehow? Anyone has any insight into this
>>>>> problem?
>>>>>
>>>>> Chen
>>>>>
>>>>>
>>>>> On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> The exception was thrown out in application master(spark streaming
>>>>>> driver) and the job shut down after this exception.
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I got the same exception after the streaming job runs for a while,
>>>>>>> The ERROR message was complaining about a temp file not being found in the
>>>>>>> output folder.
>>>>>>>
>>>>>>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming
>>>>>>> job 1407744300000 ms.0
>>>>>>> java.io.FileNotFoundException: File
>>>>>>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>>>>>>> does not exist.
>>>>>>> at
>>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>>>>>>> at
>>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>>>>>>> at
>>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>>>>>>> at
>>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>>>>>>> at
>>>>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>>>> at
>>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>>>>>>> at
>>>>>>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>>>>>>> at
>>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>>>>>>> at
>>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>>>>>>> at
>>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>>>>>>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>>>> at scala.util.Try$.apply(Try.scala:161)
>>>>>>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>>>>>>> at
>>>>>>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <
>>>>>>> bill.jaypeterson@gmail.com> wrote:
>>>>>>>
>>>>>>>> I just saw another error after my job was run for 2 hours:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>>>>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>>>>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>>>>>>
>>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>>>>>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> All my jobs use the same parameter to the function checkpoint. Is
>>>>>>>> it the reason for the error?
>>>>>>>>
>>>>>>>> I will post the stack trace of the other error after it appears
>>>>>>>> again. Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>> Bill
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>>>>>>> tathagata.das1565@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Can you give a stack trace and logs of the exception? Its hard to
>>>>>>>>> say anything without any associated stack trace and logs.
>>>>>>>>>
>>>>>>>>> TD
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <
>>>>>>>>> bill.jaypeterson@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to
>>>>>>>>>> save results into hdfs files. However, it has an exception after 20 batches
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> When the job is running, I do not change any file in the folder.
>>>>>>>>>> Does anyone know why the file cannot be found?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Bill
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chen Song
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chen Song
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chen Song
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>
>>
>
--
Chen Song
Re: saveAsTextFiles file not found exception
Posted by Andrew Ash <an...@andrewash.com>.
Hi Chen,
Please see the bug I filed at
https://issues.apache.org/jira/browse/SPARK-2984 with the
FileNotFoundException on _temporary directory issue.
Andrew
On Mon, Aug 11, 2014 at 10:50 PM, Andrew Ash <an...@andrewash.com> wrote:
> Not sure which stalled HDFS client issue your'e referring to, but there
> was one fixed in Spark 1.0.2 that could help you out --
> https://github.com/apache/spark/pull/1409. I've still seen one related
> to Configuration objects not being threadsafe though so you'd still need to
> keep speculation on to fix that (SPARK-2546)
>
> As it stands now, I can:
>
> A) have speculation off, in which case I get random hangs for a variety of
> reasons (your HDFS stall, my Configuration safety issue)
>
> or
>
> B) have speculation on, in which case I get random failures related to
> LeaseExpiredExceptions and .../_temporary/... file doesn't exist exceptions.
>
>
> Kind of a catch-22 -- there's no reliable way to run large jobs on Spark
> right now!
>
> I'm going to file a bug for the _temporary and LeaseExpiredExceptions as I
> think these are widespread enough that we need a place to track a
> resolution.
>
>
> On Mon, Aug 11, 2014 at 9:08 AM, Chen Song <ch...@gmail.com> wrote:
>
>> Andrew that is a good finding.
>>
>> Yes, I have speculative execution turned on, becauseI saw tasks stalled
>> on HDFS client.
>>
>> If I turned off speculative execution, is there a way to circumvent the
>> hanging task issue?
>>
>>
>>
>> On Mon, Aug 11, 2014 at 11:13 AM, Andrew Ash <an...@andrewash.com>
>> wrote:
>>
>>> I've also been seeing similar stacktraces on Spark core (not streaming)
>>> and have a theory it's related to spark.speculation being turned on. Do
>>> you have that enabled by chance?
>>>
>>>
>>> On Mon, Aug 11, 2014 at 8:10 AM, Chen Song <ch...@gmail.com>
>>> wrote:
>>>
>>>> Bill
>>>>
>>>> Did you get this resolved somehow? Anyone has any insight into this
>>>> problem?
>>>>
>>>> Chen
>>>>
>>>>
>>>> On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> The exception was thrown out in application master(spark streaming
>>>>> driver) and the job shut down after this exception.
>>>>>
>>>>>
>>>>> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I got the same exception after the streaming job runs for a while,
>>>>>> The ERROR message was complaining about a temp file not being found in the
>>>>>> output folder.
>>>>>>
>>>>>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
>>>>>> 1407744300000 ms.0
>>>>>> java.io.FileNotFoundException: File
>>>>>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>>>>>> does not exist.
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>>>>>> at
>>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>>>>>> at
>>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>>>>>> at
>>>>>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>>>>>> at
>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>>>>>> at
>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>>>>>> at
>>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>>>>>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>>>>>> at
>>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>>>>>> at
>>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>>>>>> at
>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>>>>>> at
>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>>> at
>>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>>> at scala.util.Try$.apply(Try.scala:161)
>>>>>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>>>>>> at
>>>>>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bill.jaypeterson@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I just saw another error after my job was run for 2 hours:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>>>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>>>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>>>>>
>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>>>>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All my jobs use the same parameter to the function checkpoint. Is it
>>>>>>> the reason for the error?
>>>>>>>
>>>>>>> I will post the stack trace of the other error after it appears
>>>>>>> again. Thanks!
>>>>>>>
>>>>>>>
>>>>>>> Bill
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>>>>>> tathagata.das1565@gmail.com> wrote:
>>>>>>>
>>>>>>>> Can you give a stack trace and logs of the exception? Its hard to
>>>>>>>> say anything without any associated stack trace and logs.
>>>>>>>>
>>>>>>>> TD
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <
>>>>>>>> bill.jaypeterson@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to
>>>>>>>>> save results into hdfs files. However, it has an exception after 20 batches
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> When the job is running, I do not change any file in the folder.
>>>>>>>>> Does anyone know why the file cannot be found?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Bill
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chen Song
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chen Song
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Chen Song
>>>>
>>>>
>>>
>>
>>
>> --
>> Chen Song
>>
>>
>
Re: saveAsTextFiles file not found exception
Posted by Andrew Ash <an...@andrewash.com>.
Not sure which stalled HDFS client issue your'e referring to, but there was
one fixed in Spark 1.0.2 that could help you out --
https://github.com/apache/spark/pull/1409. I've still seen one related to
Configuration objects not being threadsafe though so you'd still need to
keep speculation on to fix that (SPARK-2546)
As it stands now, I can:
A) have speculation off, in which case I get random hangs for a variety of
reasons (your HDFS stall, my Configuration safety issue)
or
B) have speculation on, in which case I get random failures related to
LeaseExpiredExceptions and .../_temporary/... file doesn't exist exceptions.
Kind of a catch-22 -- there's no reliable way to run large jobs on Spark
right now!
I'm going to file a bug for the _temporary and LeaseExpiredExceptions as I
think these are widespread enough that we need a place to track a
resolution.
On Mon, Aug 11, 2014 at 9:08 AM, Chen Song <ch...@gmail.com> wrote:
> Andrew that is a good finding.
>
> Yes, I have speculative execution turned on, becauseI saw tasks stalled on
> HDFS client.
>
> If I turned off speculative execution, is there a way to circumvent the
> hanging task issue?
>
>
>
> On Mon, Aug 11, 2014 at 11:13 AM, Andrew Ash <an...@andrewash.com> wrote:
>
>> I've also been seeing similar stacktraces on Spark core (not streaming)
>> and have a theory it's related to spark.speculation being turned on. Do
>> you have that enabled by chance?
>>
>>
>> On Mon, Aug 11, 2014 at 8:10 AM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> Bill
>>>
>>> Did you get this resolved somehow? Anyone has any insight into this
>>> problem?
>>>
>>> Chen
>>>
>>>
>>> On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com>
>>> wrote:
>>>
>>>> The exception was thrown out in application master(spark streaming
>>>> driver) and the job shut down after this exception.
>>>>
>>>>
>>>> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> I got the same exception after the streaming job runs for a while, The
>>>>> ERROR message was complaining about a temp file not being found in the
>>>>> output folder.
>>>>>
>>>>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
>>>>> 1407744300000 ms.0
>>>>> java.io.FileNotFoundException: File
>>>>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>>>>> does not exist.
>>>>> at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>>>>> at
>>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>>> at
>>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>>>>> at
>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>>>>> at
>>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>>>>> at
>>>>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>>>>> at
>>>>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>>>>> at
>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>>>>> at
>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>>>>> at
>>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>>>>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>>>>> at
>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>>>>> at
>>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>>>>> at
>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>>>>> at
>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>> at
>>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>>> at scala.util.Try$.apply(Try.scala:161)
>>>>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>>>>> at
>>>>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I just saw another error after my job was run for 2 hours:
>>>>>>
>>>>>>
>>>>>>
>>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>>>>
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>>>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>>>>
>>>>>>
>>>>>>
>>>>>> All my jobs use the same parameter to the function checkpoint. Is it
>>>>>> the reason for the error?
>>>>>>
>>>>>> I will post the stack trace of the other error after it appears
>>>>>> again. Thanks!
>>>>>>
>>>>>>
>>>>>> Bill
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>>>>> tathagata.das1565@gmail.com> wrote:
>>>>>>
>>>>>>> Can you give a stack trace and logs of the exception? Its hard to
>>>>>>> say anything without any associated stack trace and logs.
>>>>>>>
>>>>>>> TD
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <
>>>>>>> bill.jaypeterson@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to
>>>>>>>> save results into hdfs files. However, it has an exception after 20 batches
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>>>>
>>>>>>>>
>>>>>>>> When the job is running, I do not change any file in the folder.
>>>>>>>> Does anyone know why the file cannot be found?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Bill
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Chen Song
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Chen Song
>>>>
>>>>
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>
>>
>
>
> --
> Chen Song
>
>
Re: saveAsTextFiles file not found exception
Posted by Chen Song <ch...@gmail.com>.
Andrew that is a good finding.
Yes, I have speculative execution turned on, becauseI saw tasks stalled on
HDFS client.
If I turned off speculative execution, is there a way to circumvent the
hanging task issue?
On Mon, Aug 11, 2014 at 11:13 AM, Andrew Ash <an...@andrewash.com> wrote:
> I've also been seeing similar stacktraces on Spark core (not streaming)
> and have a theory it's related to spark.speculation being turned on. Do
> you have that enabled by chance?
>
>
> On Mon, Aug 11, 2014 at 8:10 AM, Chen Song <ch...@gmail.com> wrote:
>
>> Bill
>>
>> Did you get this resolved somehow? Anyone has any insight into this
>> problem?
>>
>> Chen
>>
>>
>> On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> The exception was thrown out in application master(spark streaming
>>> driver) and the job shut down after this exception.
>>>
>>>
>>> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
>>> wrote:
>>>
>>>> I got the same exception after the streaming job runs for a while, The
>>>> ERROR message was complaining about a temp file not being found in the
>>>> output folder.
>>>>
>>>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
>>>> 1407744300000 ms.0
>>>> java.io.FileNotFoundException: File
>>>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>>>> does not exist.
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>>>> at
>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>>> at
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>>>> at
>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>>>> at
>>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>>>> at
>>>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>>>> at
>>>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>>>> at
>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>>>> at
>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>>>> at
>>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>>>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>>>> at
>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>>>> at
>>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>>>> at
>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>>>> at
>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>> at
>>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>>> at scala.util.Try$.apply(Try.scala:161)
>>>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>>>> at
>>>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
>>>> wrote:
>>>>
>>>>> I just saw another error after my job was run for 2 hours:
>>>>>
>>>>>
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>>>
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>>>
>>>>>
>>>>>
>>>>> All my jobs use the same parameter to the function checkpoint. Is it
>>>>> the reason for the error?
>>>>>
>>>>> I will post the stack trace of the other error after it appears again.
>>>>> Thanks!
>>>>>
>>>>>
>>>>> Bill
>>>>>
>>>>>
>>>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>>>> tathagata.das1565@gmail.com> wrote:
>>>>>
>>>>>> Can you give a stack trace and logs of the exception? Its hard to say
>>>>>> anything without any associated stack trace and logs.
>>>>>>
>>>>>> TD
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <bill.jaypeterson@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to save
>>>>>>> results into hdfs files. However, it has an exception after 20 batches
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>>>
>>>>>>>
>>>>>>> When the job is running, I do not change any file in the folder.
>>>>>>> Does anyone know why the file cannot be found?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Bill
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Chen Song
>>>>
>>>>
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>
>>
>>
>> --
>> Chen Song
>>
>>
>
--
Chen Song
Re: saveAsTextFiles file not found exception
Posted by Andrew Ash <an...@andrewash.com>.
I've also been seeing similar stacktraces on Spark core (not streaming) and
have a theory it's related to spark.speculation being turned on. Do you
have that enabled by chance?
On Mon, Aug 11, 2014 at 8:10 AM, Chen Song <ch...@gmail.com> wrote:
> Bill
>
> Did you get this resolved somehow? Anyone has any insight into this
> problem?
>
> Chen
>
>
> On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com>
> wrote:
>
>> The exception was thrown out in application master(spark streaming
>> driver) and the job shut down after this exception.
>>
>>
>> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
>> wrote:
>>
>>> I got the same exception after the streaming job runs for a while, The
>>> ERROR message was complaining about a temp file not being found in the
>>> output folder.
>>>
>>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
>>> 1407744300000 ms.0
>>> java.io.FileNotFoundException: File
>>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>>> does not exist.
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>>> at
>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>>> at
>>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>>> at
>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>>> at
>>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>>> at
>>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>>> at
>>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>>> at
>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>>> at
>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>>> at
>>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>>> at
>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>>> at
>>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>>> at
>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>>> at
>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>> at
>>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>>> at scala.util.Try$.apply(Try.scala:161)
>>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>>> at
>>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
>>> wrote:
>>>
>>>> I just saw another error after my job was run for 2 hours:
>>>>
>>>>
>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>>
>>>>
>>>>
>>>> All my jobs use the same parameter to the function checkpoint. Is it
>>>> the reason for the error?
>>>>
>>>> I will post the stack trace of the other error after it appears again.
>>>> Thanks!
>>>>
>>>>
>>>> Bill
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>>> tathagata.das1565@gmail.com> wrote:
>>>>
>>>>> Can you give a stack trace and logs of the exception? Its hard to say
>>>>> anything without any associated stack trace and logs.
>>>>>
>>>>> TD
>>>>>
>>>>>
>>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <bi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to save
>>>>>> results into hdfs files. However, it has an exception after 20 batches
>>>>>>
>>>>>>
>>>>>>
>>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>>
>>>>>>
>>>>>> When the job is running, I do not change any file in the folder. Does
>>>>>> anyone know why the file cannot be found?
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Bill
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Chen Song
>>>
>>>
>>
>>
>> --
>> Chen Song
>>
>>
>
>
> --
> Chen Song
>
>
Re: saveAsTextFiles file not found exception
Posted by Chen Song <ch...@gmail.com>.
Bill
Did you get this resolved somehow? Anyone has any insight into this problem?
Chen
On Mon, Aug 11, 2014 at 10:30 AM, Chen Song <ch...@gmail.com> wrote:
> The exception was thrown out in application master(spark streaming driver)
> and the job shut down after this exception.
>
>
> On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com>
> wrote:
>
>> I got the same exception after the streaming job runs for a while, The
>> ERROR message was complaining about a temp file not being found in the
>> output folder.
>>
>> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
>> 1407744300000 ms.0
>> java.io.FileNotFoundException: File
>> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
>> does not exist.
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
>> at
>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
>> at
>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
>> at
>> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
>> at
>> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
>> at
>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
>> at
>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
>> at
>> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
>> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
>> at
>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
>> at
>> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
>> at
>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
>> at
>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>> at
>> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
>> at scala.util.Try$.apply(Try.scala:161)
>> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
>> at
>> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
>> wrote:
>>
>>> I just saw another error after my job was run for 2 hours:
>>>
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>>
>>>
>>>
>>> All my jobs use the same parameter to the function checkpoint. Is it the
>>> reason for the error?
>>>
>>> I will post the stack trace of the other error after it appears again.
>>> Thanks!
>>>
>>>
>>> Bill
>>>
>>>
>>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>>> tathagata.das1565@gmail.com> wrote:
>>>
>>>> Can you give a stack trace and logs of the exception? Its hard to say
>>>> anything without any associated stack trace and logs.
>>>>
>>>> TD
>>>>
>>>>
>>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <bi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am running a Spark Streaming job that uses saveAsTextFiles to save
>>>>> results into hdfs files. However, it has an exception after 20 batches
>>>>>
>>>>>
>>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>>
>>>>>
>>>>> When the job is running, I do not change any file in the folder. Does
>>>>> anyone know why the file cannot be found?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Bill
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Chen Song
>>
>>
>
>
> --
> Chen Song
>
>
--
Chen Song
Re: saveAsTextFiles file not found exception
Posted by Chen Song <ch...@gmail.com>.
The exception was thrown out in application master(spark streaming driver)
and the job shut down after this exception.
On Mon, Aug 11, 2014 at 10:29 AM, Chen Song <ch...@gmail.com> wrote:
> I got the same exception after the streaming job runs for a while, The
> ERROR message was complaining about a temp file not being found in the
> output folder.
>
> 14/08/11 08:05:08 ERROR JobScheduler: Error running job streaming job
> 1407744300000 ms.0
> java.io.FileNotFoundException: File
> hdfs://hadoopc/user/csong/output/human_bot/-1407744300000.out/_temporary/0/task_201408110805_0000_m_000007
> does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:654)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:102)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:712)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:708)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:360)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at
> org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:126)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:841)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:724)
> at
> org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:643)
> at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1068)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:773)
> at
> org.apache.spark.streaming.dstream.DStream$$anonfun$8.apply(DStream.scala:771)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:41)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at
> org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
> at scala.util.Try$.apply(Try.scala:161)
> at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
> at
> org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:172)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> On Fri, Jul 25, 2014 at 7:04 PM, Bill Jay <bi...@gmail.com>
> wrote:
>
>> I just saw another error after my job was run for 2 hours:
>>
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /apps/data/vddil/real-time/checkpoint/temp: File does not exist. Holder DFSClient_NONMAPREDUCE_327993456_13 does not have any open files.
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2946)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2766)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2674)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>> at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
>> 14/07/25 14:45:12 WARN CheckpointWriter: Error in attempt 1 of writing checkpoint to hdfs://gnosis-01-01-01.crl.samsung.com/apps/data/vddil/real-time/checkpoint/checkpoint-1406324700000
>>
>>
>>
>> All my jobs use the same parameter to the function checkpoint. Is it the
>> reason for the error?
>>
>> I will post the stack trace of the other error after it appears again.
>> Thanks!
>>
>>
>> Bill
>>
>>
>> On Fri, Jul 25, 2014 at 2:57 PM, Tathagata Das <
>> tathagata.das1565@gmail.com> wrote:
>>
>>> Can you give a stack trace and logs of the exception? Its hard to say
>>> anything without any associated stack trace and logs.
>>>
>>> TD
>>>
>>>
>>> On Fri, Jul 25, 2014 at 1:32 PM, Bill Jay <bi...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am running a Spark Streaming job that uses saveAsTextFiles to save
>>>> results into hdfs files. However, it has an exception after 20 batches
>>>>
>>>>
>>>> result-1406312340000/_temporary/0/task_201407251119_0000_m_000003 does not exist.
>>>>
>>>>
>>>> When the job is running, I do not change any file in the folder. Does
>>>> anyone know why the file cannot be found?
>>>>
>>>> Thanks!
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>
>
> --
> Chen Song
>
>
--
Chen Song