You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/20 04:55:38 UTC
[PR] [core] Compact Data Evolution column versions alongside clustering [paimon]
JingsongLi opened a new pull request, #10684:
URL: https://github.com/apache/paimon/pull/10684
With incremental clustering enabled, repeated Data Evolution column updates can accumulate even when their RowID groups do not need sorting. The same Flink/Spark maintenance job now plans ordinary compaction for complete groups that meet the file-count threshold and were not selected for clustering, then commits both kinds of work together.
Ordinary compaction preserves RowIDs, trusted clustering levels and column sequences. Sorted rewrites still get new RowIDs. Central preparation moves or materializes DVs as appropriate, and conflict/index maintenance uses only ranges that actually lose their old RowIDs. This preserves safe concurrent partial updates to compacted groups while still rejecting stale clustering and conflicting DV changes. The task serializer advances to v5 while reading v4 tasks as L0.
Validation: 12 common, 172 core, 20 Flink and 14 Spark cases passed locally with normal static checks enabled. Tests cover compact-only and mixed jobs, repeated updates, forward/reverse concurrency, optimistic retry, index coverage, DV handling, old task serialization and local/global sort ranges. Independent review completed. Final split from #10667 after #10683.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Compact Data Evolution column versions alongside clustering [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10684:
URL: https://github.com/apache/paimon/pull/10684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org