[jira] [Updated] (SPARK-13601) Invoke task failure callbacks before calling outputstream.close()

["Shixiong Zhu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-13601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shixiong Zhu updated SPARK-13601:
---------------------------------
    Fix Version/s: 1.6.2

> Invoke task failure callbacks before calling outputstream.close()
> -----------------------------------------------------------------
>
>                 Key: SPARK-13601
>                 URL: https://issues.apache.org/jira/browse/SPARK-13601
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Davies Liu
>            Assignee: Davies Liu
>             Fix For: 1.6.2, 2.0.0
>
>
> We need to submit another PR against Spark to call the task failure callbacks before Spark calls the close function on various output streams.
> For example, we need to intercept an exception and call TaskContext.markTaskFailed before calling close in the following code (in PairRDDFunctions.scala):
> {code}
>       Utils.tryWithSafeFinally {
>         while (iter.hasNext) {
>           val record = iter.next()
>           writer.write(record._1.asInstanceOf[AnyRef], record._2.asInstanceOf[AnyRef])
>           // Update bytes written metric every few records
>           maybeUpdateOutputMetrics(outputMetricsAndBytesWrittenCallback, recordsWritten)
>           recordsWritten += 1
>         }
>       } {
>         writer.close()
>       }
> {code}
> Changes to Spark should include unit tests to make sure this always work in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org