You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lawofcycles (via GitHub)" <gi...@apache.org> on 2026/02/21 14:13:05 UTC
[I] Support merge-on-read tables in changelog scans [iceberg]
lawofcycles opened a new issue, #15394:
URL: https://github.com/apache/iceberg/issues/15394
### Feature Request / Improvement
Currently, `IncrementalChangelogScan` throws `UnsupportedOperationException` when any snapshot in the scan range contains delete manifests.
https://github.com/apache/iceberg/blob/f1499bde7830a8f003995311cd2b3a3503214a81/core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java#L108-L111
This prevents changelog scans from working with merge-on-read tables. In Spark, `create_changelog_view` cannot be used for any table written with MOR.
### Background
The initial implementation of `IncrementalChangelogScan` in #5382 was scoped to CoW tables, with the intent to add delete file support in a follow-up PR. This feature was also requested in #8975.
### Proposal
Support delete files (position deletes, equality deletes, and deletion vectors) in changelog scans. The existing `DeleteFileIndex` / `DeleteLoader` infrastructure can already identify which rows are deleted by a given delete file, so the core building blocks are in place.
#### Phase 1. Delete file support for all format versions
For each snapshot in the scan range that contains delete manifests, the scan would
1. Identify newly added delete files alongside newly added/removed data files
2. Use the existing delete file merging infrastructure to determine which rows in existing data files are affected
3. Emit `DELETE` for rows removed by delete files, `INSERT` for rows in newly added data files
UPDATE detection (`UPDATE_BEFORE`/`UPDATE_AFTER` pairs) can rely on identifier columns, which is the same approach `CreateChangelogViewProcedure` already uses for carryover removal and update image computation.
#### Phase 2. Row lineage optimization for v3 tables
For v3 tables with row lineage, the changelog scan can be more efficient.
- `_last_updated_sequence_number` allows filtering to only rows changed within the scan range, avoiding full data file reads
- `_row_id` enables reliable matching of the same logical row across data files for UPDATE detection, without depending on identifier columns
Note that row lineage does not track changes made via equality deletes ([spec](https://iceberg.apache.org/spec/#row-lineage)), so Phase 1 remains necessary as the general path.
### Scope
- Phase 1 covers v2 tables (position deletes, equality deletes) and v3 tables (deletion vectors, legacy position deletes from upgraded tables)
- Phase 2 is an optimization for v3 tables with row lineage
### Query engine
Spark
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support merge-on-read tables in changelog scans [iceberg]
Posted by "lawofcycles (via GitHub)" <gi...@apache.org>.
lawofcycles commented on issue #15394:
URL: https://github.com/apache/iceberg/issues/15394#issuecomment-4527416608
@tomtongue Thank you for the suggestion. I'm open to co-work and reopened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support merge-on-read tables in changelog scans [iceberg]
Posted by "tomtongue (via GitHub)" <gi...@apache.org>.
tomtongue commented on issue #15394:
URL: https://github.com/apache/iceberg/issues/15394#issuecomment-4506077150
@lawofcycles Thanks for raising this. I'm also considering the row lineage can be used for tracking record changes. The row-lineage based ChangelogScan can support MoR type. So maybe we can collaborate this for the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support merge-on-read tables in changelog scans [iceberg]
Posted by "lawofcycles (via GitHub)" <gi...@apache.org>.
lawofcycles commented on issue #15394:
URL: https://github.com/apache/iceberg/issues/15394#issuecomment-3938876440
After further investigation, I found that #10935 is already working on adding delete file support to changelog scans, which covers the core of what this issue was proposing.
I initially thought row lineage could provide a more efficient path for v3 tables, but `_last_updated_sequence_number` and `_row_id` are row-level columns that still require reading data files, so the performance benefit over the delete file merging approach in #10935 is unclear.
Closing this as #10935 already addresses the main use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Support merge-on-read tables in changelog scans [iceberg]
Posted by "lawofcycles (via GitHub)" <gi...@apache.org>.
lawofcycles closed issue #15394: Support merge-on-read tables in changelog scans
URL: https://github.com/apache/iceberg/issues/15394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org