You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "abmo-x (via GitHub)" <gi...@apache.org> on 2023/02/15 18:53:09 UTC
[GitHub] [iceberg] abmo-x opened a new pull request, #6846: Spark 3.1 : Backport PR #6818
abmo-x opened a new pull request, #6846:
URL: https://github.com/apache/iceberg/pull/6846
Fix for https://github.com/apache/iceberg/issues/6817
When two spark application runs end up with same manifest file name of format "stage-%d-task-%d-manifest" due to same stage/task planning, the add_file procedure ends up overwriting the manifest resulting in data loss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi merged pull request #6846: Spark 3.1: add_files data loss when SNAPSHOT_ID_INHERITANCE_ENABLED
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #6846:
URL: https://github.com/apache/iceberg/pull/6846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on pull request #6846: Spark 3.1 : Backport PR #6818
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1431885827
@dramaticlly I want to leave them separate as they are already created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on pull request #6846: Spark 3.1 : Backport PR #6818
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1431890064
3.1 change was not intended in that pr, I cleaned it up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] abmo-x commented on pull request #6846: Spark 3.1: add_files data loss when SNAPSHOT_ID_INHERITANCE_ENABLED
Posted by "abmo-x (via GitHub)" <gi...@apache.org>.
abmo-x commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1435029763
@aokolnychyi fixed all the tests. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #6846: Spark 3.1 : Backport PR #6818
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1431888867
> @dramaticlly I want to leave them separate as they are already created.
I do see some [3.1 change](https://github.com/apache/iceberg/pull/6845/files#diff-af2bc22256d817e0027ad2ef2a3fc73cd25bc94083a0da3c1bad0039ed2170af) in 3.2 PR #6845 . I feel like we shall close this and move all to that and maybe update the title to be more self-explanatory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on pull request #6846: Spark 3.1 : Backport PR #6818
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1431875415
For backport do you want to put everything together in single PR? I guess #6845 and #6846 can merge into one since they are almost identical ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #6846: Spark 3.1 : Backport PR #6818
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1432520336
I see some test failures in 3.1. @abmo-x, could you check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #6846: Spark 3.1: add_files data loss when SNAPSHOT_ID_INHERITANCE_ENABLED
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #6846:
URL: https://github.com/apache/iceberg/pull/6846#issuecomment-1435257994
Thank you, @abmo-x!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org