You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Stephen0421 (via GitHub)" <gi...@apache.org> on 2026/10/22 08:37:57 UTC
[PR] [core] Support vector-store file compaction for data evolution tables [paimon]
Stephen0421 opened a new pull request, #10749:
URL: https://github.com/apache/paimon/pull/10749
### Purpose
Data evolution compaction could not compact dedicated vector-store files (`*.vector.<format>`). The `compactVector` path in `DataEvolutionCompactCoordinator` was a TODO, and `DataEvolutionNormalCompactTask` threw `UnsupportedOperationException`. Partial vector column updates therefore pile up small vector files that are never merged.
This PR adds vector-store file compaction, modeled on blob compaction:
- Add the option `vector-compaction.enabled` (default `false`). The Flink `DataEvolutionTableCompactSource` and the Spark `CompactProcedure` now read it instead of hard-coding `false`.
- Add `DataEvolutionVectorCompactTask` (`TaskType.VECTOR`). It reads its input vector files through the data evolution read path, so each vector field keeps the value from the file with the highest sequence number. It writes the current vector fields with the configured vector format and rolls output by `vector.target-file-size`. Row IDs are preserved, so deletion vectors on the normal files stay valid.
- Planning in `DataEvolutionCompactCoordinator`:
- A task always takes **all** vector-store files of its row range. Compacted files inherit the max sequence number of their inputs, so a file left out could start or stop winning rows against them.
- Vector files spanning several normal files are merged only when those normal files are merged in the same batch. Otherwise, each normal file's vector files are compacted on their own. Vector task ranges are added to the normal task's protected ranges, so a normal output never cuts through a vector output.
- A group is skipped if a current vector field covers only part of its row range. Otherwise the rewrite would store NULL for rows that never had that field.
- Fields dropped from the table are ignored in that coverage check, and the rewrite does not keep them, matching normal compaction.
- With `global-index.column-update-action=IGNORE`, compacted vector files keep per-column `writeColsSequences`, so compaction does not look like a column update to the global index. The shared logic moved from `DataEvolutionNormalCompactTask` to `DataEvolutionCompactTask`.
- `DataEvolutionCompactTaskSerializer` adds type code 3 but stays at version 4. The new type has no extra payload, and bumping the version would stop every existing job from restoring in-flight compaction tasks.
Not included: materializing deletion vectors into vector-store files.
### Tests
- `DataEvolutionCompactCoordinatorTest`:
- Planning vector tasks within one normal file and across normal files merged in the same batch.
- Skipping a field that covers only part of the range, and ignoring dropped fields.
- Serializer round trip for `VECTOR` tasks, and rejection of non-vector inputs.
- `VectorStoreTableTest`:
- Partial column updates keep the latest value per column after compaction (json and parquet), including projections and after a column rename.
- A column that is missing for part of the range is skipped.
- A dropped column is not kept.
- Output rolls by `vector.target-file-size`.
- Per-column sequences are kept under `global-index.column-update-action=IGNORE`.
- Regression: `DataEvolutionNormalCompactTaskTest`, `BlobTableTest`, `DataEvolutionTableTest`, `DataEvolutionDeletionVectorTest`, `MultipleBlobTableTest` and `CompactCandidateRangeCollectorTest` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support vector-store file compaction for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10749:
URL: https://github.com/apache/paimon/pull/10749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support vector-store file compaction for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10749:
URL: https://github.com/apache/paimon/pull/10749#discussion_r4357374770
##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java:
##########
@@ -530,6 +563,13 @@ private List<DataEvolutionCompactTask> triggerTask(
return tasks;
}
+ private boolean vectorStoreFilesCompactable(List<DataFileMeta> vectorStoreFiles) {
+ return vectorStoreFiles.size() > 1
+ && vectorStoreFiles.size() >= compactMinFileNum
+ && DataEvolutionVectorCompactTask.coversRangeForEveryField(
+ vectorStoreFiles, schemaFetcher, currentVectorFieldIds);
Review Comment:
**[P1] Make vector compaction converge after rolling its output**
This predicate also accepts the output of a successful compaction whenever the writer rolls into at least `compaction.min.file-num` vector files. With `vector-compaction.enabled=true`, Spark's `compact` procedure can therefore keep rewriting these files and committing new snapshots indefinitely: `DataEvolutionRewriteExecutor` replans the committed snapshot after each commit, `CompactProcedure` creates a new coordinator, and the completed-file protection excludes vector outputs.
I reproduced the repeated planning by extending `VectorStoreTableTest#testCompactVectorFilesRollsByVectorTargetSize`:
- Write two batches of 1,250 rows, with `compaction.min.file-num=2` and `vector.target-file-size=1 b`.
- The first compaction produces one normal file covering `[0,2499]` and three vector files covering `[0,999]`, `[1000,1999]`, and `[2000,2499]`.
- A second compaction replaces all three vector files with new filenames but identical ranges. Planning the resulting snapshot still returns another `VECTOR` task.
The core reproduction and Spark control flow establish the loop; I did not run an end-to-end Spark hang test. This is not limited to the small test target: it also occurs when an ordinary compaction range needs at least the minimum number of rolled vector outputs.
Please make selection account for target size and overlap so that already compacted, non-overlapping output layouts are skipped when no useful merge remains, while preserving the all-files-per-range correctness constraint. Tracking completed vector outputs within an invocation can additionally prevent repeated work in the same call. Please also add a regression assertion that subsequent planning produces no vector task for this layout, or a bounded Spark procedure test that verifies termination.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support vector-store file compaction for data evolution tables [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10749:
URL: https://github.com/apache/paimon/pull/10749#issuecomment-6409771747
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support vector-store file compaction for data evolution tables [paimon]
Posted by "Stephen0421 (via GitHub)" <gi...@apache.org>.
Stephen0421 commented on code in PR #10749:
URL: https://github.com/apache/paimon/pull/10749#discussion_r4360856739
##########
paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionCompactCoordinator.java:
##########
@@ -530,6 +563,13 @@ private List<DataEvolutionCompactTask> triggerTask(
return tasks;
}
+ private boolean vectorStoreFilesCompactable(List<DataFileMeta> vectorStoreFiles) {
+ return vectorStoreFiles.size() > 1
+ && vectorStoreFiles.size() >= compactMinFileNum
+ && DataEvolutionVectorCompactTask.coversRangeForEveryField(
+ vectorStoreFiles, schemaFetcher, currentVectorFieldIds);
Review Comment:
Thanks for catching this! Fixed in d948fe053d. Vector compaction now converges, and a rolled output layout is no longer replanned.
**Root cause**: `vectorStoreFilesCompactable` only checked the file count and field coverage, so an already compacted layout with at least `compaction.min.file-num` rolled files was selected again on every replan.
**Fix** (two layers, both keep the all-files-per-range constraint):
1. **Select by overlap and target size** (`DataEvolutionCompactCoordinator`): a vector group is rewritten only if some rows are stored in more than one vector file (partial column updates), or more than one file is below `vector.target-file-size`. Rolled output has no overlap and at most one small trailing file, so later invocations skip it. If a group is selected, the task still takes every vector file of its range.
2. **Track completed vector outputs within an invocation** (`CompactProcedure`), like `completedNormalFiles`: a group made up only of this invocation's vector outputs is not planned again. The size rule alone is not enough here, because several formats roll by an estimated size (e.g. Parquet's buffered data size, ORC/Vortex/Mosaic estimates), so a rolled file's final size can end up below the target. If a new file lands in the range, the whole group is still compacted together.
The Flink source plans once against a fixed snapshot and never replans its own outputs, so it is unaffected.
**Tests**:
- `VectorStoreTableTest#testCompactVectorFilesRollsByVectorTargetSize`: now asserts that planning the compacted snapshot returns no task. The input now includes a column update, because two non-overlapping files above the target are correctly skipped.
- `DataEvolutionCompactCoordinatorTest`: new cases for skipping rolled output, merging only when overlapping or with at least two small files, and skipping completed vector outputs unless a new file joins the range.
- New bounded Spark procedure test `data evolution vector compaction does not recompact rolled output`: the commit callback fails if the procedure commits more than once. It fails on the previous code, because the procedure tried a second commit that rewrote its own output, and it passes with this fix, with one commit and the expected three rolled vector files.
I also updated `data-evolution-maintenance.mdx` to describe when a vector group is skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org