You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rdblue (via GitHub)" <gi...@apache.org> on 2026/09/24 19:21:05 UTC
[PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
rdblue opened a new pull request, #18248:
URL: https://github.com/apache/iceberg/pull/18248
This implements MDV filtering in V4ManifestReader.
The MDV filter is run after inheritance and first row ID assignment because first row ID assignment must be done for all files. The filter is also run before more expensive filtering, like content stats and partition filtering. If no DV is present, only status filtering is enabled to avoid checking whether the MDV is null on every record.
This also updates the `includeAll` mode, which returns records that have `DELETED` or `REPLACED` status in the file. When `includeAll` encounters a file that is live in the manifest and deleted by its MDV, it sets the status to `DELETED` and the snapshot ID to null because the actual snapshot ID when the file was deleted is not known. This is strange because there are no other situations in which a manifest entry contains a null snapshot ID.
I think the right fix for the null snapshot ID issue is not to have an `includeAll` mode, but since that may be controversial I think we should fix this when we are implementing reads to detect changes. When reading a manifest to find changes, the reader needs the manifest's replaced and deleted positions from `Tracking` that have a known snapshot ID (`dv_snapshot_id`, if it matches the snapshot for which changes are being scanned).
Test plan:
- TestTrackingStruct:
- Test `convertToDeleted` and `convertToReplaced` methods modify only status and snapshot ID
- TestV4ManfiestReader:
- Validate that MDV filtering removes live files and is a no-op for non-live files
- Validate that `includeAll` returns MDV-deleted files as `DELETED` with a null `snapshot_id` (see above)
- Validate that `includeAll` does not modify already deleted records
- Update inheritance, first row ID, and MDV filtering tests to use full validation of `Tracking`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4098889193
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -318,49 +321,143 @@ public void readManifestFile(FileFormat format) throws IOException {
@FieldSource("MANIFEST_FORMATS")
public void statusFilter(FileFormat format) throws IOException {
List<TrackedFile> files =
- ImmutableList.of(
- unpartitionedFileWithStatus(EntryStatus.ADDED, "s3://bucket/added.parquet"),
- unpartitionedFileWithStatus(EntryStatus.MODIFIED, "s3://bucket/modified.parquet"),
- unpartitionedFileWithStatus(EntryStatus.DELETED, "s3://bucket/deleted.parquet"),
- unpartitionedFileWithStatus(EntryStatus.EXISTING, "s3://bucket/existing.parquet"),
- unpartitionedFileWithStatus(EntryStatus.REPLACED, "s3://bucket/replaced.parquet"));
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+ when(manifest.firstRowId()).thenReturn(10_000L);
- List<TrackedFile> liveFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG));
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ V4ManifestReader.Builder builder =
+ V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
+ .metricsConfig(METRICS_CONFIG);
+
+ List<TrackedFile> liveFiles = read(builder);
assertThat(liveFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactly(files.get(0), files.get(1), files.get(3));
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .containsExactly(expectedFiles.get(0), expectedFiles.get(1), expectedFiles.get(3));
- List<TrackedFile> allFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG)
- .includeAll());
+ List<TrackedFile> allFiles = read(builder.includeAll());
assertThat(allFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactlyElementsOf(files);
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expectedFiles);
+ }
+
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void mdvFilter(FileFormat format) throws IOException {
+ List<TrackedFile> files =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+
+ // this bitmap deletes a MODIFIED entry and an already DELETED entry
+ when(manifest.manifestDeletionVector())
+ .thenReturn(MumblingTestUtil.bitmap(MumblingTestUtil.sparse(1, 2)));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile( // status changed to DELETED, delete snapshot ID is unknown
Review Comment:
got it. this is for `includeAll` scenario where manifest DV marked entries are converted to deleted status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader, remove includeAll [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #18248:
URL: https://github.com/apache/iceberg/pull/18248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4099194022
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -113,11 +115,20 @@ Schema readSchema() {
@Override
public CloseableIterator<TrackedFile> iterator() {
+ // first row ID assignment must happen first to maintain consistent assignment when files are
+ // removed or filtered
CloseableIterable<TrackedFile> files =
CloseableIterable.transform(open(), this::applyInheritance);
if (!includeAll) {
- files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ if (dv != null) {
+ files =
+ CloseableIterable.filter(files, file -> file.tracking().isLive() && !isDeleted(file));
+ } else {
+ files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ }
+ } else if (dv != null) {
+ files = CloseableIterable.transform(files, this::applyMDVDeletes);
Review Comment:
actually I would favor `failing` for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4108373176
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -156,6 +167,20 @@ private TrackedFile applyInheritance(TrackedFile file) {
return file;
}
+ private boolean isDeleted(TrackedFile file) {
Review Comment:
I updated this to `isDeletedByMDV`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4097931939
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -113,11 +115,20 @@ Schema readSchema() {
@Override
public CloseableIterator<TrackedFile> iterator() {
+ // first row ID assignment must happen first to maintain consistent assignment when files are
+ // removed or filtered
CloseableIterable<TrackedFile> files =
CloseableIterable.transform(open(), this::applyInheritance);
if (!includeAll) {
- files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ if (dv != null) {
+ files =
+ CloseableIterable.filter(files, file -> file.tracking().isLive() && !isDeleted(file));
+ } else {
+ files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ }
+ } else if (dv != null) {
+ files = CloseableIterable.transform(files, this::applyMDVDeletes);
Review Comment:
Thanks for the context in the PR description.
I am wondering if we should just fail here (instead of returning partially correct result)? or remove includeAll for now (like you mentioned) until we have the proper support for change detection read?
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -156,6 +167,20 @@ private TrackedFile applyInheritance(TrackedFile file) {
return file;
}
+ private boolean isDeleted(TrackedFile file) {
Review Comment:
nit: my initial reaction of this method name is the status `deleted`. should we change it to `isPositionDeleted`?
##########
core/src/main/java/org/apache/iceberg/TrackingStruct.java:
##########
@@ -179,6 +180,32 @@ boolean assignFirstRowId(Long nextRowId) {
return false;
}
+ /**
+ * Update this tracking to have DELETED status.
+ *
+ * @param snapshotId snapshot ID when the record was deleted, or null if it is unknown
+ * @return this converted to a DELETED Tracking
+ */
+ TrackingStruct convertToDeleted(Long snapshotId) {
+ Preconditions.checkState(status.isLive(), "Cannot convert %s to DELETED", status);
+ this.status = EntryStatus.DELETED;
+ this.snapshotId = snapshotId;
+ return this;
+ }
+
+ /**
+ * Update this tracking to have REPLACED status.
+ *
+ * @param snapshotId snapshot ID when the record was replaced, or null if it is unknown
+ * @return this converted to a REPLACED Tracking
+ */
+ TrackingStruct convertToReplaced(Long snapshotId) {
Review Comment:
this method is not used in this PR.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -370,146 +467,178 @@ public void inheritanceUncommittedOnlyInheritsSnapshotId(FileFormat format) thro
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
+ new TrackingStruct(EntryStatus.ADDED, 34L, null, null, null, null, null, null),
+ "s3://bucket/table/file-a.parquet"));
+
assertThat(actual)
- .extracting(file -> file.tracking().snapshotId())
- .containsExactly(1234567L, SNAPSHOT_ID);
- assertThat(actual)
- .as("Uncommitted files do not inherit data sequence number")
- .extracting(file -> file.tracking().dataSequenceNumber())
- .containsOnlyNulls();
- assertThat(actual)
- .as("Uncommitted files do not inherit file sequence number")
- .extracting(file -> file.tracking().fileSequenceNumber())
- .containsOnlyNulls();
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expected);
}
@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
public void inheritanceSnapshotId(FileFormat format) throws IOException {
- Tracking trackingWithSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null);
TrackedFile withSnapshotId =
- unpartitionedDataFile(trackingWithSnapshotId, "s3://bucket/table/file-b.parquet");
- Tracking trackingWithoutSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, null, null, null, null, null, null, null);
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, 5L, 5L, null, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet");
TrackedFile withoutSnapshotId =
- unpartitionedDataFile(trackingWithoutSnapshotId, "s3://bucket/table/file-a.parquet");
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, null, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet");
ManifestFile manifest =
writeManifest(
format, UNPARTITIONED_TYPE, ImmutableList.of(withSnapshotId, withoutSnapshotId));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+ when(manifest.snapshotId()).thenReturn(34L);
+
V4ManifestReader.Builder builder =
V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, ID_PARTITIONING_SPECS)
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, 5L, 5L, null, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
+ new TrackingStruct(EntryStatus.ADDED, 34L, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet"));
+
assertThat(actual)
- .extracting(file -> file.tracking().snapshotId())
- .containsExactly(1234567L, SNAPSHOT_ID);
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expected);
}
@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
public void inheritanceDVSnapshotIdNotInherited(FileFormat format) throws IOException {
- Tracking trackingWithDVSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, null, null, 1234567L, null, null, null);
TrackedFile withDVSnapshotId =
- unpartitionedDataFile(trackingWithDVSnapshotId, "s3://bucket/table/file-b.parquet");
- Tracking trackingWithoutDVSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, null, null, null, null, null, null);
+ unpartitionedDataFile(
+ new TrackingStruct(
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet");
TrackedFile withoutDVSnapshotId =
- unpartitionedDataFile(trackingWithoutDVSnapshotId, "s3://bucket/table/file-a.parquet");
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet");
ManifestFile manifest =
writeManifest(
format, UNPARTITIONED_TYPE, ImmutableList.of(withDVSnapshotId, withoutDVSnapshotId));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+ when(manifest.snapshotId()).thenReturn(34L);
+
V4ManifestReader.Builder builder =
V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, ID_PARTITIONING_SPECS)
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
Review Comment:
nit: the comment probably should be `// dv_snapshot_id stays null (not inherited)`
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -318,49 +321,143 @@ public void readManifestFile(FileFormat format) throws IOException {
@FieldSource("MANIFEST_FORMATS")
public void statusFilter(FileFormat format) throws IOException {
List<TrackedFile> files =
- ImmutableList.of(
- unpartitionedFileWithStatus(EntryStatus.ADDED, "s3://bucket/added.parquet"),
- unpartitionedFileWithStatus(EntryStatus.MODIFIED, "s3://bucket/modified.parquet"),
- unpartitionedFileWithStatus(EntryStatus.DELETED, "s3://bucket/deleted.parquet"),
- unpartitionedFileWithStatus(EntryStatus.EXISTING, "s3://bucket/existing.parquet"),
- unpartitionedFileWithStatus(EntryStatus.REPLACED, "s3://bucket/replaced.parquet"));
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+ when(manifest.firstRowId()).thenReturn(10_000L);
- List<TrackedFile> liveFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG));
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ V4ManifestReader.Builder builder =
+ V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
+ .metricsConfig(METRICS_CONFIG);
+
+ List<TrackedFile> liveFiles = read(builder);
assertThat(liveFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactly(files.get(0), files.get(1), files.get(3));
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .containsExactly(expectedFiles.get(0), expectedFiles.get(1), expectedFiles.get(3));
- List<TrackedFile> allFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG)
- .includeAll());
+ List<TrackedFile> allFiles = read(builder.includeAll());
assertThat(allFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactlyElementsOf(files);
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expectedFiles);
+ }
+
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void mdvFilter(FileFormat format) throws IOException {
+ List<TrackedFile> files =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+
+ // this bitmap deletes a MODIFIED entry and an already DELETED entry
+ when(manifest.manifestDeletionVector())
+ .thenReturn(MumblingTestUtil.bitmap(MumblingTestUtil.sparse(1, 2)));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile( // status changed to DELETED, delete snapshot ID is unknown
Review Comment:
why is the modified entry converted to deleted?
##########
core/src/test/java/org/apache/iceberg/V4TestComparators.java:
##########
@@ -94,9 +94,7 @@ private static <T extends Comparable<T>> Comparator<T> natural() {
.thenComparing(Tracking::dvSnapshotId, natural())
.thenComparing(Tracking::firstRowId, natural())
.thenComparing(Tracking::deletedPositions, BYTES)
- .thenComparing(Tracking::replacedPositions, BYTES)
- .thenComparing(Tracking::manifestLocation, natural())
- .thenComparingLong(Tracking::manifestPos));
+ .thenComparing(Tracking::replacedPositions, BYTES));
Review Comment:
should we add the context as inline code comment for future reference on why the two fields are not included in the comparator construction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4108378738
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -370,146 +467,178 @@ public void inheritanceUncommittedOnlyInheritsSnapshotId(FileFormat format) thro
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
+ new TrackingStruct(EntryStatus.ADDED, 34L, null, null, null, null, null, null),
+ "s3://bucket/table/file-a.parquet"));
+
assertThat(actual)
- .extracting(file -> file.tracking().snapshotId())
- .containsExactly(1234567L, SNAPSHOT_ID);
- assertThat(actual)
- .as("Uncommitted files do not inherit data sequence number")
- .extracting(file -> file.tracking().dataSequenceNumber())
- .containsOnlyNulls();
- assertThat(actual)
- .as("Uncommitted files do not inherit file sequence number")
- .extracting(file -> file.tracking().fileSequenceNumber())
- .containsOnlyNulls();
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expected);
}
@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
public void inheritanceSnapshotId(FileFormat format) throws IOException {
- Tracking trackingWithSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null);
TrackedFile withSnapshotId =
- unpartitionedDataFile(trackingWithSnapshotId, "s3://bucket/table/file-b.parquet");
- Tracking trackingWithoutSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, null, null, null, null, null, null, null);
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, 5L, 5L, null, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet");
TrackedFile withoutSnapshotId =
- unpartitionedDataFile(trackingWithoutSnapshotId, "s3://bucket/table/file-a.parquet");
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, null, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet");
ManifestFile manifest =
writeManifest(
format, UNPARTITIONED_TYPE, ImmutableList.of(withSnapshotId, withoutSnapshotId));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+ when(manifest.snapshotId()).thenReturn(34L);
+
V4ManifestReader.Builder builder =
V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, ID_PARTITIONING_SPECS)
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, 5L, 5L, null, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
+ new TrackingStruct(EntryStatus.ADDED, 34L, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet"));
+
assertThat(actual)
- .extracting(file -> file.tracking().snapshotId())
- .containsExactly(1234567L, SNAPSHOT_ID);
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expected);
}
@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
public void inheritanceDVSnapshotIdNotInherited(FileFormat format) throws IOException {
- Tracking trackingWithDVSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, null, null, 1234567L, null, null, null);
TrackedFile withDVSnapshotId =
- unpartitionedDataFile(trackingWithDVSnapshotId, "s3://bucket/table/file-b.parquet");
- Tracking trackingWithoutDVSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, null, null, null, null, null, null);
+ unpartitionedDataFile(
+ new TrackingStruct(
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet");
TrackedFile withoutDVSnapshotId =
- unpartitionedDataFile(trackingWithoutDVSnapshotId, "s3://bucket/table/file-a.parquet");
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_100L, null, null),
+ "s3://bucket/table/file-a.parquet");
ManifestFile manifest =
writeManifest(
format, UNPARTITIONED_TYPE, ImmutableList.of(withDVSnapshotId, withoutDVSnapshotId));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+ when(manifest.snapshotId()).thenReturn(34L);
+
V4ManifestReader.Builder builder =
V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, ID_PARTITIONING_SPECS)
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4098541714
##########
core/src/main/java/org/apache/iceberg/TrackingStruct.java:
##########
@@ -179,6 +180,32 @@ boolean assignFirstRowId(Long nextRowId) {
return false;
}
+ /**
+ * Update this tracking to have DELETED status.
+ *
+ * @param snapshotId snapshot ID when the record was deleted, or null if it is unknown
+ * @return this converted to a DELETED Tracking
+ */
+ TrackingStruct convertToDeleted(Long snapshotId) {
+ Preconditions.checkState(status.isLive(), "Cannot convert %s to DELETED", status);
+ this.status = EntryStatus.DELETED;
+ this.snapshotId = snapshotId;
+ return this;
+ }
+
+ /**
+ * Update this tracking to have REPLACED status.
+ *
+ * @param snapshotId snapshot ID when the record was replaced, or null if it is unknown
+ * @return this converted to a REPLACED Tracking
+ */
+ TrackingStruct convertToReplaced(Long snapshotId) {
Review Comment:
This is going to be needed when we produce replaced entries from the replaced bitmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4097633134
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -370,146 +467,178 @@ public void inheritanceUncommittedOnlyInheritsSnapshotId(FileFormat format) thro
.metricsConfig(METRICS_CONFIG);
List<TrackedFile> actual = read(builder);
+ List<TrackedFile> expected =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null),
+ "s3://bucket/table/file-b.parquet"),
+ unpartitionedDataFile( // inherits only snapshot ID
+ new TrackingStruct(EntryStatus.ADDED, 34L, null, null, null, null, null, null),
+ "s3://bucket/table/file-a.parquet"));
+
assertThat(actual)
- .extracting(file -> file.tracking().snapshotId())
- .containsExactly(1234567L, SNAPSHOT_ID);
- assertThat(actual)
- .as("Uncommitted files do not inherit data sequence number")
- .extracting(file -> file.tracking().dataSequenceNumber())
- .containsOnlyNulls();
- assertThat(actual)
- .as("Uncommitted files do not inherit file sequence number")
- .extracting(file -> file.tracking().fileSequenceNumber())
- .containsOnlyNulls();
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expected);
}
@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
public void inheritanceSnapshotId(FileFormat format) throws IOException {
- Tracking trackingWithSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, 1234567L, null, null, null, null, null, null);
TrackedFile withSnapshotId =
- unpartitionedDataFile(trackingWithSnapshotId, "s3://bucket/table/file-b.parquet");
- Tracking trackingWithoutSnapshotId =
- new TrackingStruct(EntryStatus.ADDED, null, null, null, null, null, null, null);
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 1234567L, 5L, 5L, null, 5_000L, null, null),
+ "s3://bucket/table/file-b.parquet");
TrackedFile withoutSnapshotId =
- unpartitionedDataFile(trackingWithoutSnapshotId, "s3://bucket/table/file-a.parquet");
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, null, 5L, 5L, null, 5_100L, null, null),
Review Comment:
Existing tests were updated to use constants rather than null values to isolate specific inheritance cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4108375715
##########
core/src/main/java/org/apache/iceberg/TrackingStruct.java:
##########
@@ -179,6 +180,32 @@ boolean assignFirstRowId(Long nextRowId) {
return false;
}
+ /**
+ * Update this tracking to have DELETED status.
+ *
+ * @param snapshotId snapshot ID when the record was deleted, or null if it is unknown
+ * @return this converted to a DELETED Tracking
+ */
+ TrackingStruct convertToDeleted(Long snapshotId) {
+ Preconditions.checkState(status.isLive(), "Cannot convert %s to DELETED", status);
+ this.status = EntryStatus.DELETED;
+ this.snapshotId = snapshotId;
+ return this;
+ }
+
+ /**
+ * Update this tracking to have REPLACED status.
+ *
+ * @param snapshotId snapshot ID when the record was replaced, or null if it is unknown
+ * @return this converted to a REPLACED Tracking
+ */
+ TrackingStruct convertToReplaced(Long snapshotId) {
Review Comment:
Removed, which was part of removing `includeAll`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4108387875
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -1702,20 +1852,6 @@ private static TrackedFile unpartitionedDataFile(Tracking tracking, String locat
return unpartitionedDataFile(tracking, location, null /* no stats */, null /* no DV */);
}
- private static TrackedFile unpartitionedFileWithStatus(EntryStatus status, String location) {
Review Comment:
This is no longer used now that we are comparing all `Tracking` fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4098548395
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -318,49 +321,143 @@ public void readManifestFile(FileFormat format) throws IOException {
@FieldSource("MANIFEST_FORMATS")
public void statusFilter(FileFormat format) throws IOException {
List<TrackedFile> files =
- ImmutableList.of(
- unpartitionedFileWithStatus(EntryStatus.ADDED, "s3://bucket/added.parquet"),
- unpartitionedFileWithStatus(EntryStatus.MODIFIED, "s3://bucket/modified.parquet"),
- unpartitionedFileWithStatus(EntryStatus.DELETED, "s3://bucket/deleted.parquet"),
- unpartitionedFileWithStatus(EntryStatus.EXISTING, "s3://bucket/existing.parquet"),
- unpartitionedFileWithStatus(EntryStatus.REPLACED, "s3://bucket/replaced.parquet"));
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+ when(manifest.firstRowId()).thenReturn(10_000L);
- List<TrackedFile> liveFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG));
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ V4ManifestReader.Builder builder =
+ V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
+ .metricsConfig(METRICS_CONFIG);
+
+ List<TrackedFile> liveFiles = read(builder);
assertThat(liveFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactly(files.get(0), files.get(1), files.get(3));
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .containsExactly(expectedFiles.get(0), expectedFiles.get(1), expectedFiles.get(3));
- List<TrackedFile> allFiles =
- read(
- V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA, UNPARTITIONED_SPECS)
- .metricsConfig(METRICS_CONFIG)
- .includeAll());
+ List<TrackedFile> allFiles = read(builder.includeAll());
assertThat(allFiles)
- .usingComparatorForType(FILE_COMPARATOR, TrackedFile.class)
- .containsExactlyElementsOf(files);
+ .usingComparatorForType(FILE_AND_TRACKING_COMPARATOR, TrackedFile.class)
+ .isEqualTo(expectedFiles);
+ }
+
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void mdvFilter(FileFormat format) throws IOException {
+ List<TrackedFile> files =
+ List.of(
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.ADDED, 42L, null, null, null, null, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.MODIFIED, 40L, 5L, 5L, 42L, 5_000L, null, null),
+ "s3://bucket/modified.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.DELETED, 42L, 2L, 2L, null, 1_000L, null, null),
+ "s3://bucket/deleted.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.EXISTING, 38L, 4L, 4L, null, 3_000L, null, null),
+ "s3://bucket/existing.parquet"),
+ unpartitionedDataFile(
+ new TrackingStruct(EntryStatus.REPLACED, 42L, 2L, 2L, 40L, 2_000L, null, null),
+ "s3://bucket/replaced.parquet"));
+
+ ManifestFile manifest = writeManifest(format, UNPARTITIONED_TYPE, files);
+
+ // this bitmap deletes a MODIFIED entry and an already DELETED entry
+ when(manifest.manifestDeletionVector())
+ .thenReturn(MumblingTestUtil.bitmap(MumblingTestUtil.sparse(1, 2)));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+
+ List<TrackedFile> expectedFiles =
+ List.of(
+ unpartitionedDataFile( // inherits seq number and assigned first row ID
+ new TrackingStruct(
+ EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null, 10_000L, null, null),
+ "s3://bucket/added.parquet"),
+ unpartitionedDataFile( // status changed to DELETED, delete snapshot ID is unknown
Review Comment:
It was deleted by the Manifest Deletion Vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4098552889
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -113,11 +115,20 @@ Schema readSchema() {
@Override
public CloseableIterator<TrackedFile> iterator() {
+ // first row ID assignment must happen first to maintain consistent assignment when files are
+ // removed or filtered
CloseableIterable<TrackedFile> files =
CloseableIterable.transform(open(), this::applyInheritance);
if (!includeAll) {
- files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ if (dv != null) {
+ files =
+ CloseableIterable.filter(files, file -> file.tracking().isLive() && !isDeleted(file));
+ } else {
+ files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ }
+ } else if (dv != null) {
+ files = CloseableIterable.transform(files, this::applyMDVDeletes);
Review Comment:
I'm fine with that. I just wanted to make sure there was agreement before I removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4099194022
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -113,11 +115,20 @@ Schema readSchema() {
@Override
public CloseableIterator<TrackedFile> iterator() {
+ // first row ID assignment must happen first to maintain consistent assignment when files are
+ // removed or filtered
CloseableIterable<TrackedFile> files =
CloseableIterable.transform(open(), this::applyInheritance);
if (!includeAll) {
- files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ if (dv != null) {
+ files =
+ CloseableIterable.filter(files, file -> file.tracking().isLive() && !isDeleted(file));
+ } else {
+ files = CloseableIterable.filter(files, file -> file.tracking().isLive());
+ }
+ } else if (dv != null) {
+ files = CloseableIterable.transform(files, this::applyMDVDeletes);
Review Comment:
actually I would favor `failing` (than removing `includeAll`) for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4108391909
##########
core/src/test/java/org/apache/iceberg/V4TestComparators.java:
##########
@@ -94,9 +94,7 @@ private static <T extends Comparable<T>> Comparator<T> natural() {
.thenComparing(Tracking::dvSnapshotId, natural())
.thenComparing(Tracking::firstRowId, natural())
.thenComparing(Tracking::deletedPositions, BYTES)
- .thenComparing(Tracking::replacedPositions, BYTES)
- .thenComparing(Tracking::manifestLocation, natural())
- .thenComparingLong(Tracking::manifestPos));
+ .thenComparing(Tracking::replacedPositions, BYTES));
Review Comment:
I don't think so. It's more confusing that they were checked, I think. It took me a bit to figure out why tests were failing even though all of the data in the failure message matched!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4097617106
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -117,6 +118,8 @@ class TestV4ManifestReader {
V4TestComparators.trackedFileStatusOnly(ID_PARTITIONED_TYPE);
private static final Comparator<TrackedFile> UNPARTITIONED_FILE_COMPARATOR =
V4TestComparators.trackedFileStatusOnly(UNPARTITIONED_TYPE);
+ private static final Comparator<TrackedFile> FILE_AND_TRACKING_COMPARATOR =
+ V4TestComparators.trackedFile(UNPARTITIONED_TYPE);
Review Comment:
The majority of the changes to this file update inheritance and first row ID assignment tests to validate the entire `Tracking` struct, like these tests do for `TrackedFile` using the `FILE_COMPARATOR`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support manifest DVs in V4ManifestReader [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #18248:
URL: https://github.com/apache/iceberg/pull/18248#discussion_r4097608005
##########
core/src/test/java/org/apache/iceberg/V4TestComparators.java:
##########
@@ -94,9 +94,7 @@ private static <T extends Comparable<T>> Comparator<T> natural() {
.thenComparing(Tracking::dvSnapshotId, natural())
.thenComparing(Tracking::firstRowId, natural())
.thenComparing(Tracking::deletedPositions, BYTES)
- .thenComparing(Tracking::replacedPositions, BYTES)
- .thenComparing(Tracking::manifestLocation, natural())
- .thenComparingLong(Tracking::manifestPos));
+ .thenComparing(Tracking::replacedPositions, BYTES));
Review Comment:
This was needed because the inheritance tests don't need to validate the _file or _pos fields, and the inheritance test for _file and _pos just verifies the metadata columns and doesn't use this comparator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org