You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "haizhou-zhao (via GitHub)" <gi...@apache.org> on 2024/06/18 20:29:17 UTC
[I] [DeleteManifest] Making file validation optional [iceberg]
haizhou-zhao opened a new issue, #10535:
URL: https://github.com/apache/iceberg/issues/10535
### Feature Request / Improvement
## Background
This issue comes up when we are trying to fix a corrupted table where one manifest file (with active reference to data files) was (unintentionally) removed from the file system. At that point, our only way out was to remove the reference to that manifest file so that queries on the table does not fail with NotFoundException. This are the APIs we executed to achieve that:
```
table.rewriteManifests().deleteManifest(manifest).commit()
```
However, we found that Manifest rewrite implementation will force consistency on file counts, which means the above line of code will fail due to the following validation:
https://github.com/apache/iceberg/blob/c7de6cb345995cb47312edbef6edae2f17fb8aba/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L182
## Feature Request
Although keeping file counts consistent with manifest rewrite serves most cases, in corner cases like fixing a corrupted table with manifest file deleted, user does intentionally want to lose all the data files referred by that missing manifest file (so that query against the table could start to work, and they can backfill the lost data later). We'd like to raise a feature request to make `validateFilesCounts` check optional on manifest rewrite, so that users have the choice to disable it by some configuration.
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2176943839
@aokolnychyi @szehon-ho I am wondering what do you think.
Basically the problem we are trying to resolve is remove existing manifests from table and I hope to achieve this with extra configuration like `forced()` in `table.rewriteManifests().deleteManifest(manifest).forced().commit()` so that we can skip the validation on active file count.
The reason to remove existing manifest can vary, but for our use case we are trying to recover from failure in #8599 , where REST client prematurely removed manifests on network timeout but the commit actual did go through on catalog side. So table ends up in a corrupted state where manifest file is deleted on file system but tracked in iceberg metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2177253344
Well, that was my thought as its parallel with appendManifest. But now I see https://github.com/apache/iceberg/pull/10396 try to deprecate that, so like to get @Fokko thought here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2571807148
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2558674207
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2176984560
btw, did you look at that case? I didnt follow but from #8599 supoosedly it only happens when its instance of CleanableException
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "haizhou-zhao (via GitHub)" <gi...@apache.org>.
haizhou-zhao commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2176916293
cc: @dramaticlly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2176968134
I guess they initially made rewriteManifest as a safe API.
I wonder, does DeleteFiles API achieve this? The next commit will remove that manifest if all its contents are removed.
If we really need a specific API, it sounds like we need deleteManifest method there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #10535:
URL: https://github.com/apache/iceberg/issues/10535#issuecomment-2177054658
thank you Szehon, I think `DeleteFiles` API only work with data files, delete files but not for manifest files. I tried to leverage delete by file path in [DeleteFiles](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/DeleteFiles.java#L45) ->[MergingSnapshotProducer](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L187-L202) but it does not work.
I agreed that deleteManifest sounds like a right way to do this instead of stub in rewriteManifests. I am aware of the fix 8599 above, it's just that some damage has occurred this time and we are looking forward to a way to fix corrupted table. There can be potentially other reasons for dropping a manifest from iceberg metadata, which is not relate to 8599.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] [DeleteManifest] Making file validation optional [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #10535: [DeleteManifest] Making file validation optional
URL: https://github.com/apache/iceberg/issues/10535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org