[jira] [Assigned] (SPARK-25331) Structured Streaming File Sink duplicates records in case of driver failure

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-25331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-25331:
------------------------------------

    Assignee:     (was: Apache Spark)

> Structured Streaming File Sink duplicates records in case of driver failure
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-25331
>                 URL: https://issues.apache.org/jira/browse/SPARK-25331
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.3.1
>            Reporter: Mihaly Toth
>            Priority: Major
>
> Lets assume {{FileStreamSink.addBtach}} is called and an appropriate job has been started by {{FileFormatWrite.write}} and then the resulting task sets are completed but in the meantime the driver dies. In such a case repeating {{FileStreamSink.addBtach}} will result in duplicate writing of the data
> In the event the driver fails after the executors start processing the job the processed batch will be written twice.
> Steps needed:
> # call {{FileStreamSink.addBtach}}
> # make the {{ManifestFileCommitProtocol}} fail to finish its {{commitJob}}
> # call {{FileStreamSink.addBtach}} with the same data
> # make the {{ManifestFileCommitProtocol}} finish its {{commitJob}} successfully
> # Verify file output - according to {{Sink.addBatch}} documentation the rdd should be written only once
> I have created a wip PR with a unit test:
> https://github.com/apache/spark/pull/22331



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org