You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2017/01/19 02:13:26 UTC

[jira] [Resolved] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

     [ https://issues.apache.org/jira/browse/SPARK-19183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-19183.
---------------------------------
       Resolution: Fixed
         Assignee: Eric Liang
    Fix Version/s: 2.2.0

> Add deleteWithJob hook to internal commit protocol API
> ------------------------------------------------------
>
>                 Key: SPARK-19183
>                 URL: https://issues.apache.org/jira/browse/SPARK-19183
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Eric Liang
>            Assignee: Eric Liang
>             Fix For: 2.2.0
>
>
> Currently in SQL we implement overwrites by calling fs.delete() directly on the original data. This is not ideal since we the original files end up deleted even if the job aborts. We should extend the commit protocol to allow file overwrites to be managed as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org