You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ferrari6666 (via GitHub)" <gi...@apache.org> on 2023/03/02 17:20:07 UTC
[GitHub] [iceberg] ferrari6666 opened a new issue, #6989: Table Corruption iceberg
ferrari6666 opened a new issue, #6989:
URL: https://github.com/apache/iceberg/issues/6989
### Query engine
Spark
### Question
When spark abort is called from SparkWrite if few files are deleted but manifest,manifestLists and metadata not deleted this is causing table corruption with AWS glue catalog TableOperations Pls can throw some light on this.
Catalog commit was in UNKOWN state which causes cleanupOnAbort = false; ideally this should not attempt file cleanup.
It attempted for file cleanup as S3 logs shows file marked as deleted.
(edited)
[8:30](https://apache-iceberg.slack.com/archives/C03LG1D563F/p1677769243280359)
if (deletedFilesCount.get() < paths.size()) {
LOG.warn("Deleted only {} of {} file(s) ({})", deletedFilesCount, paths.size(), context);
}
[8:31](https://apache-iceberg.slack.com/archives/C03LG1D563F/p1677769291365349)
manual rollback needs to be done?
@Anton
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Table Corruption iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6989:
URL: https://github.com/apache/iceberg/issues/6989#issuecomment-1879443537
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #6989: Table Corruption iceberg
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #6989:
URL: https://github.com/apache/iceberg/issues/6989#issuecomment-1500286398
Can you please be a little more clear about what happened? Did the metadata.json from the commit get changed in the catalog? Were the Aborted files deleted? Did Spark treat the write as failed?
What versions of Iceberg and Spark are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Table Corruption iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6989:
URL: https://github.com/apache/iceberg/issues/6989#issuecomment-1868144249
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ferrari6666 commented on issue #6989: Table Corruption iceberg
Posted by "ferrari6666 (via GitHub)" <gi...@apache.org>.
ferrari6666 commented on issue #6989:
URL: https://github.com/apache/iceberg/issues/6989#issuecomment-1500101644
@jackye1995 pls can you look into this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Table Corruption iceberg [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6989: Table Corruption iceberg
URL: https://github.com/apache/iceberg/issues/6989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org