You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hari Shreedharan (JIRA)" <ji...@apache.org> on 2015/02/03 03:04:34 UTC

[jira] [Created] (SPARK-5545) [STREAMING] DStream#saveAs**Files can fail after app restarts

Hari Shreedharan created SPARK-5545:
---------------------------------------

             Summary: [STREAMING] DStream#saveAs**Files can fail after app restarts
                 Key: SPARK-5545
                 URL: https://issues.apache.org/jira/browse/SPARK-5545
             Project: Spark
          Issue Type: Bug
            Reporter: Hari Shreedharan


After an app restarts, sometimes the saveAs**Files can fail. This happens if the driver dies while the RDD was being written to HDFS. At this point the rdd-<timestamp> directory has already been created but we have not marked it as completely processed. This causes the RDD to get written after we restart, into the same directory. This can cause the underlying MR api to throw an exception that looks like this:
{code}
15/02/02 13:16:41 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Output directory hdfs://wypoon-cdhx-1.ent.cloudera.com:8020/user/systest/flumetest/rdd-1422911774000 already exists)
Exception in thread "Driver" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://wypoon-cdhx-1.ent.cloudera.com:8020/user/systest/flumetest/rdd-1422911774000 already exists
	at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1041)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:940)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:849)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1164)
...
{code}

Thanks to [~wypoon] for finding this issue!

I have a PR coming up for this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org