You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "nickdelnano (via GitHub)" <gi...@apache.org> on 2025/08/20 18:37:16 UTC

[I] [Bug] [core] Iceberg: createMetadataWithoutBase only includes latest Paimon snapshot [paimon]

nickdelnano opened a new issue, #6107:
URL: https://github.com/apache/paimon/issues/6107

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   Code path:  https://github.com/apache/paimon/blob/ff02f6bf3ceccf8dcac38bc58cf6db390509bd46/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java#L305C49-L305C59
   
   createMetadataWithoutBase is called under certain conditions like
   - enabling Iceberg compatibility for the first time
   - committing Iceberg metadata after a previous commit failed in the Iceberg layer (for many possible reasons)
   
   In some cases the Paimon table will have many snapshots already (e.g. snapshots 1 to 1000) and after calling createMetadataWithoutBase, only the latest Paimon commit will be synced to Iceberg.
   
   By syncing the whole Paimon history to Iceberg, the Iceberg compatibility feature becomes more reliable and suitable for production use cases that require Iceberg time travel.
   
   My use case is this:
   - Sync MySQL tables to Paimon using Flink CDC
   - Tag daily Paimon snapshots [automatically](https://paimon.apache.org/docs/master/maintenance/manage-tags/)
   - Iceberg readers read daily snapshots
   
   Upon any failure in the Iceberg committer, the metadata needs to be recreated. This pipeline breaks and cannot be recovered.
   
   Consider making this feature opt-in with configuration as it may be a costly operation to sync many Paimon snapshots to Iceberg and therefore reach the Flink checkpoint timeout
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] [core] Iceberg: createMetadataWithoutBase should include full history of Paimon snapshots [paimon]

Posted by "nickdelnano (via GitHub)" <gi...@apache.org>.
nickdelnano commented on issue #6107:
URL: https://github.com/apache/paimon/issues/6107#issuecomment-3208130226

   @LsomeYeah what do you think about this issue and the suggested solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] [core] Iceberg: createMetadataWithoutBase should include full history of Paimon snapshots [paimon]

Posted by "LsomeYeah (via GitHub)" <gi...@apache.org>.
LsomeYeah commented on issue #6107:
URL: https://github.com/apache/paimon/issues/6107#issuecomment-3208725719

   > [@LsomeYeah](https://github.com/LsomeYeah) what do you think about this issue and the suggested solution?
   
   @nickdelnano I think this case is reasonable, and I also prefer adding a new option.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] [core] Iceberg: createMetadataWithoutBase should include full history of Paimon snapshots [paimon]

Posted by "nickdelnano (via GitHub)" <gi...@apache.org>.
nickdelnano commented on issue #6107:
URL: https://github.com/apache/paimon/issues/6107#issuecomment-3235697293

   Investigated this some. Syncing all Paimon snapshots to Iceberg looks possible. I realized I have another requirement in my use case around Paimon tags and Iceberg compatibility.
   
   More details on my use case:
   
   * Sync MySQL tables to Paimon using Flink CDC
   * Tag daily Paimon snapshots [automatically](https://paimon.apache.org/docs/master/maintenance/manage-tags/)
   * Iceberg readers read daily snapshots
   
   Table configuration
   `snapshot.time-retained: 7d`
   `tag.num-retained-max: 100`
   `tag.automatic-creation: watermark`
   `tag.creation-period: daily`
   
   After `snapshot.time-retained` snapshots are expired from Paimon but daily tags still exist. I need all tags available in Iceberg. `IcebergCommitCallback` only considers snapshots in the Paimon table so this does not work. I will check on this next.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org