You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (JIRA)" <ji...@apache.org> on 2019/03/22 22:44:00 UTC

[jira] [Created] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

Jungtaek Lim created SPARK-27254:
------------------------------------

             Summary: Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted
                 Key: SPARK-27254
                 URL: https://issues.apache.org/jira/browse/SPARK-27254
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Jungtaek Lim


ManifestFileCommitProtocol doesn't clean up complete (but will become invalid) output files when job is aborted.

ManifestFileCommitProtocol doesn't do anything for cleaning up when job is aborted but just maintains the metadata which list of complete output files are written. SPARK-27210 addressed for task level cleanup, but it still doesn't clean up it as job level.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org