[GitHub] [incubator-iceberg] rdblue commented on issue #698: [Spark-3] Add missing abortStagedChanges support for StagedSparkTable

[GitBox <gi...@apache.org>]

rdblue commented on issue #698: [Spark-3] Add missing abortStagedChanges support for StagedSparkTable
URL: https://github.com/apache/incubator-iceberg/issues/698#issuecomment-569363867
 
 
   @jerryshao, I think that we do need to implement the abort method. The abort is called if there is any failure with the final commit (see https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala#L494).
   
   That commit happens after the write operation has committed to the transaction, so Spark is no longer responsible for calling the writer's abort methods with each task's commit data. If the transaction has a successful write committed, then Spark won't clean up the data files that were created and Iceberg should. It's a narrow case, where the write succeeds and Iceberg's commit fails, but we should plan on handling it. Because it is such a narrow case, I didn't think it was very urgent.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org