You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/08/21 04:06:10 UTC
[PR] [core] Optimize conflict scan for index-only commits [paimon]
leaves12138 opened a new pull request, #9334:
URL: https://github.com/apache/paimon/pull/9334
### Purpose
Optimize conflict detection for Data Evolution commits that append only global index files. These commits now scan the row ranges referenced by the index files instead of loading every data file in the changed partitions.
Keep index-only overwrite commits on the overwrite-specific path so deletion-vector references are still scanned correctly.
### Tests
- `mvn -pl paimon-core -DwildcardSuites=none -Dtest=ConflictDetectionTest test`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimize data evolution conflict scans [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9334:
URL: https://github.com/apache/paimon/pull/9334#discussion_r3827525554
##########
paimon-core/src/main/java/org/apache/paimon/operation/commit/DataEvolutionConflictDetection.java:
##########
@@ -220,14 +174,7 @@ private List<SimpleFileEntry> scanOverwriteBaseDataFiles(
.readAllEntriesFromDataFiles(
latestSnapshot, changedPartitions, referencedDataFiles);
}
- return super.scanBaseDataFiles(
- latestSnapshot,
- changedPartitions,
- deltaFiles,
- indexFiles,
- CommitKind.OVERWRITE,
- previousAttempt,
- hasOverwriteSincePreviousAttempt);
+ return Collections.emptyList();
Review Comment:
**[P1] Preserve explicit APPEND conflict scans**
This empty fallback also applies when recovery explicitly requests `checkAppendFiles=true`. Ordinary Data Evolution APPEND files still have `firstRowId == null` at this point because row tracking is assigned later, so they provide neither a row-range selector nor a referenced filename. Replaying an already committed committable is therefore accepted instead of being compared with the live base file. In an end-to-end reproduction, the recovery commit advanced the snapshot and the same physical row was returned twice; the base branch rejects the replay with `Trying to add file ... already in the map`.
Please propagate the explicit append-file conflict-check requirement separately from `discardDuplicate` and retain the changed-partition scan for non-empty APPEND deltas when that check is requested. Index-only APPENDs can keep the selective path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Optimize data evolution conflict scans [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9334:
URL: https://github.com/apache/paimon/pull/9334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org