You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/23 05:53:41 UTC

[GitHub] [iceberg] singhpk234 opened a new pull request, #4836: Spark: Extend CommitStateUnknown exception handling in Spark Actions

singhpk234 opened a new pull request, #4836:
URL: https://github.com/apache/iceberg/pull/4836

   ### About the Change
   
   This is an extension of the work done in https://github.com/apache/iceberg/pull/4687
   
   Spark actions also manipulate the table obj, apart of SparkWrite and hence, the handling of CommitStateUnknown exception when encountered, should not delete the added files.
   
   While actions such as RewriteDatafiles have this handling :
   https://github.com/apache/iceberg/blob/566b2fe3f31f4a53f5a26aca1c4239085de4a994/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java#L108-L120
   
   found this missing in RewriteManifestSparkAction, hence added the same. 
   
   ----- 
   
   ### Testing done
   
   Added a UT to demonstrate the E2E , without the change the UT fails with File Not found exception thus leading to table corruption 
   
   ---- 
   
   cc @RussellSpitzer, @stevenzwu @flyrain @aokolnychyi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] singhpk234 commented on pull request #4836: Spark: Extend CommitStateUnknown exception handling in Spark Actions

Posted by GitBox <gi...@apache.org>.
singhpk234 commented on PR #4836:
URL: https://github.com/apache/iceberg/pull/4836#issuecomment-1135383263

   Sure thing!!, Added a [PR](https://github.com/apache/iceberg/pull/4850) for backporting this to (3.1 / 3.0 / 2.4).
   
   Thanks @RussellSpitzer  !!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] RussellSpitzer commented on pull request #4836: Spark: Extend CommitStateUnknown exception handling in Spark Actions

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on PR #4836:
URL: https://github.com/apache/iceberg/pull/4836#issuecomment-1134957923

   Let's be sure to backport this to 3.1 and 3.0 as well (2.4 may also be vulnerable?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] ajantha-bhat commented on pull request #4836: Spark: Extend CommitStateUnknown exception handling in Spark Actions

Posted by GitBox <gi...@apache.org>.
ajantha-bhat commented on PR #4836:
URL: https://github.com/apache/iceberg/pull/4836#issuecomment-1135391008

   @nastra : We need to take this for 0.13.X branch as well for 0.13.2 release I guess. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

[GitHub] [iceberg] RussellSpitzer merged pull request #4836: Spark: Extend CommitStateUnknown exception handling in Spark Actions

Posted by GitBox <gi...@apache.org>.
RussellSpitzer merged PR #4836:
URL: https://github.com/apache/iceberg/pull/4836


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org