You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JunRuiLee (via GitHub)" <gi...@apache.org> on 2026/05/25 02:36:08 UTC
[PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
JunRuiLee opened a new pull request, #7948:
URL: https://github.com/apache/paimon/pull/7948
## Motivation
In some production shared-dataset deployments, compaction is separated from writers and runs as dedicated jobs. For DV primary-key tables, batch scans skip level-0 files by default, so newly written uncompacted data is not visible until compaction finishes. This means readers depend on compaction progress to see the latest data.
`visibility-callback.enabled` does not fit this requirement because it also relies on compaction: commits are returned only after compaction makes the data visible through the optimized read path. In this scenario, users want to read the latest committed data without waiting for compaction.
## Changes
This PR adds a new option `deletion-vectors.merge-on-read` for DV tables. When enabled, batch scans include DV level-0 files and merge them at read time, so readers can see uncompacted committed data without depending on compaction. This trades read performance for fresher batch results.
This option only affects batch scan visibility of DV level-0 files. It does not change streaming scan or changelog behavior.
- Add `deletion-vectors.merge-on-read` core option (default `false`).
- Use it to control whether DV batch scans skip level-0 files (`CoreOptions.batchScanSkipLevel0()`).
- Validate mutual exclusion with `visibility-callback.enabled` in `SchemaValidation`.
- Add core-level tests for batch read behavior with and without merge-on-read.
- Add Flink IT test for merge-on-read with write-only mode.
- Update generated config documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3295841733
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -620,6 +620,13 @@ private static void validateForDeletionVectors(CoreOptions options) {
!options.mergeEngine().equals(MergeEngine.FIRST_ROW),
"First row merge engine does not need deletion vectors because there is no deletion of old data in this merge engine.");
}
+
+ checkArgument(
+ !(options.toConfiguration().get(CoreOptions.DELETION_VECTORS_MERGE_ON_READ)
+ && options.visibilityCallbackEnabled()),
Review Comment:
**Consider validating `merge-on-read` requires DV enabled**
`validateForDeletionVectors` is only called when `deletionVectorsEnabled()` is true. If a user sets `deletion-vectors.merge-on-read=true` without `deletion-vectors.enabled=true`, the option is silently ignored. This could be a confusing misconfiguration.
Consider adding a check (either here or in the general validation section):
```java
checkArgument(
!(options.toConfiguration().get(CoreOptions.DELETION_VECTORS_MERGE_ON_READ)
&& !options.deletionVectorsEnabled()),
"deletion-vectors.merge-on-read requires deletion-vectors.enabled to be true.");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3295841360
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3649,7 +3660,10 @@ public boolean forceLookup() {
Review Comment:
**Suggestion: add a helper accessor method**
For consistency with other boolean options (e.g., `deletionVectorsEnabled()`, `visibilityCallbackEnabled()`), consider adding a dedicated accessor method:
```java
public boolean deletionVectorsMergeOnRead() {
return options.get(DELETION_VECTORS_MERGE_ON_READ);
}
```
Then `batchScanSkipLevel0()` can call `deletionVectorsMergeOnRead()` instead of directly accessing the config, and `SchemaValidation` can also use `options.deletionVectorsMergeOnRead()` instead of `options.toConfiguration().get(CoreOptions.DELETION_VECTORS_MERGE_ON_READ)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3297269404
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -620,6 +624,12 @@ private static void validateForDeletionVectors(CoreOptions options) {
!options.mergeEngine().equals(MergeEngine.FIRST_ROW),
"First row merge engine does not need deletion vectors because there is no deletion of old data in this merge engine.");
}
+
+ checkArgument(
+ !(options.deletionVectorsMergeOnRead() && options.visibilityCallbackEnabled()),
Review Comment:
Good point. Removed the conflict check. Added positive validation tests and extended `VisibilityCallbackTest` to cover `(dv=true, postpone=true)` with `merge-on-read=true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JunRuiLee (via GitHub)" <gi...@apache.org>.
JunRuiLee commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3295887779
##########
paimon-api/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -3649,7 +3660,10 @@ public boolean forceLookup() {
Review Comment:
Good call, added `deletionVectorsMergeOnRead()` and updated both callers.
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -620,6 +620,13 @@ private static void validateForDeletionVectors(CoreOptions options) {
!options.mergeEngine().equals(MergeEngine.FIRST_ROW),
"First row merge engine does not need deletion vectors because there is no deletion of old data in this merge engine.");
}
+
+ checkArgument(
+ !(options.toConfiguration().get(CoreOptions.DELETION_VECTORS_MERGE_ON_READ)
+ && options.visibilityCallbackEnabled()),
Review Comment:
Makes sense. Added a check in the general validation section so it fails early with a clear message. Also added a test for this case.
##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/DeletionVectorITCase.java:
##########
@@ -206,6 +206,38 @@ public void testBatchReadDVTable(String changelogProducer, boolean dvBitmap64) {
Row.of(1, "111111111"), Row.of(2, "2_1"), Row.of(3, "3_1"), Row.of(4, "4"));
}
+ @ParameterizedTest
+ @MethodSource("parameters1")
+ public void testBatchReadDVTableWithMergeOnRead(String changelogProducer, boolean dvBitmap64) {
+ sql(
+ String.format(
+ "CREATE TABLE T (id INT PRIMARY KEY NOT ENFORCED, name STRING) "
+ + "WITH ('deletion-vectors.enabled' = 'true', 'changelog-producer' = '%s', "
+ + "'deletion-vectors.bitmap64' = '%s', 'write-only' = 'true')",
+ changelogProducer, dvBitmap64));
+
+ sql("INSERT INTO T VALUES (1, '111111111'), (2, '2'), (3, '3'), (4, '4')");
+
+ sql("INSERT INTO T VALUES (2, '2_1'), (3, '3_1')");
+
Review Comment:
Fixed — added `'bucket' = '1'` to align with the core test, and changed the assertion to `isEmpty()` for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3296913707
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java:
##########
@@ -620,6 +624,12 @@ private static void validateForDeletionVectors(CoreOptions options) {
!options.mergeEngine().equals(MergeEngine.FIRST_ROW),
"First row merge engine does not need deletion vectors because there is no deletion of old data in this merge engine.");
}
+
+ checkArgument(
+ !(options.deletionVectorsMergeOnRead() && options.visibilityCallbackEnabled()),
Review Comment:
visibilityCallbackEnabled also works for postpone bucket. So it is meaningful to enable deletion-vectors.merge-on-read together with visibility-callback.enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #7948:
URL: https://github.com/apache/paimon/pull/7948#discussion_r3295842268
##########
paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/DeletionVectorITCase.java:
##########
@@ -206,6 +206,38 @@ public void testBatchReadDVTable(String changelogProducer, boolean dvBitmap64) {
Row.of(1, "111111111"), Row.of(2, "2_1"), Row.of(3, "3_1"), Row.of(4, "4"));
}
+ @ParameterizedTest
+ @MethodSource("parameters1")
+ public void testBatchReadDVTableWithMergeOnRead(String changelogProducer, boolean dvBitmap64) {
+ sql(
+ String.format(
+ "CREATE TABLE T (id INT PRIMARY KEY NOT ENFORCED, name STRING) "
+ + "WITH ('deletion-vectors.enabled' = 'true', 'changelog-producer' = '%s', "
+ + "'deletion-vectors.bitmap64' = '%s', 'write-only' = 'true')",
+ changelogProducer, dvBitmap64));
+
+ sql("INSERT INTO T VALUES (1, '111111111'), (2, '2'), (3, '3'), (4, '4')");
+
+ sql("INSERT INTO T VALUES (2, '2_1'), (3, '3_1')");
+
Review Comment:
**Inconsistency with core test `testDefaultSkipsLevel0`**
The core-level test `DeletionVectorsMergeOnReadTest#testDefaultSkipsLevel0` asserts an **empty** result when reading without merge-on-read (all files at level 0, none visible). But this Flink IT test asserts the **first commit's data is visible** without merge-on-read.
This difference is likely because the core test explicitly sets `bucket=1` (fixed bucket) while this Flink test doesn't (defaults to dynamic bucket mode, which may place initial files differently). Could be worth:
1. Adding a brief comment explaining why the first commit's data is visible here (e.g., files from the first snapshot are at level > 0 in this config), or
2. Aligning the two tests to use the same bucket configuration for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Add merge-on-read option for DV table batch L0 visibility [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7948:
URL: https://github.com/apache/paimon/pull/7948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org