You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "amoghnatu (via GitHub)" <gi...@apache.org> on 2025/12/26 23:28:42 UTC
[I] Handle log truncation in Delta to Iceberg incremental sync [incubator-xtable]
amoghnatu opened a new issue, #779:
URL: https://github.com/apache/incubator-xtable/issues/779
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
### Please describe the bug 🐞
There appears to be a discrepancy between how incremental‑sync safety is validated for Delta vs. Iceberg when historical metadata has been truncated. Iceberg correctly detects expired snapshots and triggers a full snapshot sync, but Delta’s `isIncrementalSyncSafeFrom` only checks whether a commit existed at the given instant—not whether the underlying log files still exist. As a result, incremental sync is incorrectly marked “safe,” leading to failures such as `DeltaFileNotFoundException` when attempting to reconstruct old commits. This issue proposes enhancing Delta’s safety check to verify that required commit files are still available before approving incremental sync.
### Are you willing to submit PR?
- [x] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Handle log truncation in Delta to Iceberg incremental sync [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on issue #779:
URL: https://github.com/apache/incubator-xtable/issues/779#issuecomment-3739345155
I have a draft PR for a fallback check.
https://github.com/apache/incubator-xtable/pull/782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Handle log truncation in Delta to Iceberg incremental sync [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on issue #779:
URL: https://github.com/apache/incubator-xtable/issues/779#issuecomment-3696975119
@amoghnatu Thanks for reporting the issue, do you have a unit test by any chance to reproduce the issue? After that, we can improve the this check to get that test passed.
https://github.com/apache/incubator-xtable/blob/main/xtable-core/src/main/java/org/apache/xtable/delta/DeltaConversionSource.java#L201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org