You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lucasgameiroborges (via GitHub)" <gi...@apache.org> on 2026/08/21 09:46:19 UTC

[I] [Bug] IcebergCommitCallback.expireAllBefore fails permanently on a manifest list a previous rebuild already deleted [paimon]

lucasgameiroborges opened a new issue, #9342:
URL: https://github.com/apache/paimon/issues/9342

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   1.4.1 (but code seems identical on current `master` `4a346f450`)
   
   ### Compute Engine
   
   Flink 2.0
   
   ### Minimal reproduce step
   
   Unit test for reproduction will be included in the PR. Not sure we can reliably reproduce in a live system but the process was:
   
   1. Writing continuously to a Paimon table with Iceberg REST metadata sync enabled.
   2. Get a committer recovery that replays a committable more than one snapshot behind the catalog, ie `createMetadataWithoutBase` runs.
   3. `expireAllBefore(N)` deletes the manifest lists of every version < N. Then at some point we need to have `v(N-1).metadata.json` retained while the manifest lists it references have just been deleted.
   4. Again get to a committer recovery where createMetadataWithoutBase runs. `expireAllBefore` now walks `v(N-1).metadata.json` and calls `manifestList.read(listName)` on a list that was already deleted in step 3. This will lead to:
   
   ```
   java.lang.RuntimeException: Failed to read snap-1-1f878b70-445c-4332-98a4-73ea0c549437.avro
       at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:131)
       at org.apache.paimon.iceberg.IcebergCommitCallback.expireAllBefore(IcebergCommitCallback.java:985)
       at org.apache.paimon.iceberg.IcebergCommitCallback.createMetadataWithoutBase(IcebergCommitCallback.java:386)
       at org.apache.paimon.iceberg.IcebergCommitCallback.retry(IcebergCommitCallback.java:249)
       at org.apache.paimon.operation.FileStoreCommitImpl.filterCommitted(FileStoreCommitImpl.java:277)
       at org.apache.paimon.flink.sink.RestoreCommittableStateManager.recover(RestoreCommittableStateManager.java:81)
       at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:147)
   Caused by: java.io.FileNotFoundException: File '…/metadata/snap-1-1f878b70-….avro' not found
   ```
   
   ### What doesn't meet your expectations?
   
   `expireAllBefore` exists to delete the manifests and manifest lists of expired versions. It deletes them with `deleteQuietly` (tolerating absence) but reads the manifest list with a hard `manifestList.read(...)` that throws when the file is already gone. If the two cleanup steps lead to a retained metadata JSON referencing an already-deleted list, the task dies during Flink state initialisation, the committable is never dropped, and recovery replays it until the retained JSON happens to be deleted by later churn.
   
   Skipping a manifest list that no longer exists seems safe since the loop's only purpose in reading it is to delete what it references, and those manifests were deleted by the same earlier pass.
   
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] IcebergCommitCallback.expireAllBefore fails permanently on a manifest list a previous rebuild already deleted [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #9342: [Bug] IcebergCommitCallback.expireAllBefore fails permanently on a manifest list a previous rebuild already deleted
URL: https://github.com/apache/paimon/issues/9342


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org