You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anoopj (via GitHub)" <gi...@apache.org> on 2025/11/08 00:07:50 UTC
[PR] [WIP] V4 Manifest Read Support [iceberg]
anoopj opened a new pull request, #14533:
URL: https://github.com/apache/iceberg/pull/14533
WIP PR for [s.apache.org/iceberg-single-file-commit](https://s.apache.org/iceberg-single-file-commit)
Implemented so far:
* Foundational types such as TrackedFile interface as unified representation for all V4 entry types
* Reader and basic root manifest expansion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520509160
##########
api/src/main/java/org/apache/iceberg/DeletionVector.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Deletion vector information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This encapsulates both inline and out-of-line deletion vector content. This struct must be
+ * defined for types 1 (POSITION_DELETES) and 5 (MANIFEST_DV).
+ */
+public interface DeletionVector {
+ Types.NestedField OFFSET =
+ Types.NestedField.optional(
+ 144, "offset", Types.LongType.get(), "Offset in the file where the content starts");
+ Types.NestedField SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 145,
+ "size_in_bytes",
+ Types.LongType.get(),
+ "Length of a referenced content stored in the file; required if offset is present");
+ Types.NestedField INLINE_CONTENT =
+ Types.NestedField.optional(
+ 146, "inline_content", Types.BinaryType.get(), "Serialized bitmap for inline DVs");
+
+ /**
+ * Returns the offset in the file where the deletion vector content starts.
+ *
+ * <p>This is used for out-of-line deletion vectors stored in Puffin files.
+ */
+ Long offset();
+
+ /**
+ * Returns the size in bytes of the deletion vector content.
+ *
+ * <p>Required if offset is present.
+ */
+ Long sizeInBytes();
+
+ /**
+ * Returns the serialized bitmap for inline deletion vectors.
+ *
+ * <p>When present, the deletion vector is stored inline in the manifest rather than in a separate
+ * Puffin file.
+ */
+ ByteBuffer inlineContent();
Review Comment:
I mentioned this in my comment below, but I don't think there's much value in combining the inline MDV metadata and fields to track data DVs stored in Puffin. These aren't overlapping, so I'd keep them separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539903760
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadataFactory.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/** Factory for creating {@link InheritableTrackedMetadata} instances. */
+class InheritableTrackedMetadataFactory {
+
+ private InheritableTrackedMetadataFactory() {}
+
+ /**
+ * Creates inheritable metadata from explicit values.
+ *
+ * @param snapshotId the snapshot ID
+ * @param sequenceNumber the sequence number
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata create(long snapshotId, long sequenceNumber) {
+ return new BaseInheritableTrackedMetadata(snapshotId, sequenceNumber);
+ }
+
+ /**
+ * Creates inheritable metadata from a tracked file representing a manifest (for reading leaf
+ * manifests).
+ *
+ * @param manifestEntry the DATA_MANIFEST or DELETE_MANIFEST tracked file from root
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata fromTrackedFile(TrackedFile<?> manifestEntry) {
+ Preconditions.checkArgument(
+ manifestEntry.contentType() == FileContent.DATA_MANIFEST
+ || manifestEntry.contentType() == FileContent.DELETE_MANIFEST,
+ "Can only create metadata from manifest entries, got: %s",
+ manifestEntry.contentType());
+
+ TrackingInfo tracking = manifestEntry.trackingInfo();
+ Long snapshotId = tracking != null ? tracking.snapshotId() : null;
+ Long sequenceNumber = tracking != null ? tracking.sequenceNumber() : null;
+
+ Preconditions.checkArgument(
+ snapshotId != null, "Manifest entry must have snapshot ID: %s", manifestEntry.location());
Review Comment:
The term "manifest entry" is confusing in v4 because it could be referring to an entry in a manifest (the original definition) or a tracked file for a manifest. This is why we are using the term "tracked file" now. Method names, variable names, and error messages for v4 metadata should always replace "manifest entry" with a more specific and less ambiguous term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539932750
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
+ }
+
+ @Override
+ public Long pos() {
+ return trackedFile.pos();
+ }
+
+ @Override
+ public int specId() {
+ return trackedFile.partitionSpecId();
+ }
+
+ @Override
+ public FileContent content() {
+ return trackedFile.contentType();
+ }
+
+ @Override
+ public CharSequence path() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public String location() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public FileFormat format() {
+ return trackedFile.fileFormat();
+ }
+
+ @Override
+ public StructLike partition() {
+ return null;
+ }
+
+ @Override
+ public long recordCount() {
+ return trackedFile.recordCount();
+ }
+
+ @Override
+ public long fileSizeInBytes() {
+ Long size = trackedFile.fileSizeInBytes();
+ return size != null ? size : 0L;
+ }
+
+ @Override
+ public Map<Integer, Long> columnSizes() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> valueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> nullValueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> nanValueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, ByteBuffer> lowerBounds() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, ByteBuffer> upperBounds() {
+ return null;
+ }
+
+ @Override
+ public ByteBuffer keyMetadata() {
+ return trackedFile.keyMetadata();
+ }
+
+ @Override
+ public List<Long> splitOffsets() {
+ return trackedFile.splitOffsets();
+ }
+
+ @Override
+ public List<Integer> equalityFieldIds() {
+ return null;
+ }
+
+ @Override
+ public Integer sortOrderId() {
+ return trackedFile.sortOrderId();
+ }
+
+ @Override
+ public Long dataSequenceNumber() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.sequenceNumber() : null;
+ }
+
+ @Override
+ public Long fileSequenceNumber() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.fileSequenceNumber() : null;
+ }
+
+ @Override
+ public Long firstRowId() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.firstRowId() : null;
+ }
+
+ @Override
+ public DataFile copy() {
+ return new TrackedDataFileAdapter((TrackedFile<?>) trackedFile.copy(), spec);
+ }
+
+ @Override
+ public DataFile copyWithoutStats() {
+ return new TrackedDataFileAdapter((TrackedFile<?>) trackedFile.copyWithoutStats(), spec);
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> requestedColumnIds) {
+ return new TrackedDataFileAdapter(
+ (TrackedFile<?>) trackedFile.copyWithStats(requestedColumnIds), spec);
Review Comment:
Why is a cast necessary here? This is suspicious; probably caused by the type param on `TrackedFile`. Hopefully we can remove this when we fix that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539925754
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
+ }
+
+ @Override
+ public Long pos() {
+ return trackedFile.pos();
+ }
+
+ @Override
+ public int specId() {
+ return trackedFile.partitionSpecId();
+ }
+
+ @Override
+ public FileContent content() {
+ return trackedFile.contentType();
+ }
+
+ @Override
+ public CharSequence path() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public String location() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public FileFormat format() {
+ return trackedFile.fileFormat();
+ }
+
+ @Override
+ public StructLike partition() {
+ return null;
+ }
+
+ @Override
+ public long recordCount() {
+ return trackedFile.recordCount();
+ }
+
+ @Override
+ public long fileSizeInBytes() {
+ Long size = trackedFile.fileSizeInBytes();
+ return size != null ? size : 0L;
Review Comment:
Iceberg should never lie about metadata. Instead, allow this to throw a null pointer exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2687985857
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
Review Comment:
That sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2688277363
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
Review Comment:
Yep, exactly what the code does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683244203
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
Review Comment:
Moved them into `ContentInfoStruct`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2529460196
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520543212
##########
core/src/main/java/org/apache/iceberg/DataTableScan.java:
##########
@@ -63,8 +63,17 @@ protected TableScan newRefinedScan(Table table, Schema schema, TableScanContext
@Override
public CloseableIterable<FileScanTask> doPlanFiles() {
Snapshot snapshot = snapshot();
-
Review Comment:
Nit: Avoid unnecessary whitespace changes. They cause conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539834543
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
Review Comment:
Other field docs expand the enum symbols: `"Entry status: 0=existing, 1=added, 2=deleted/removed"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539881385
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
Review Comment:
I don't think there's much value in parameterizing `TrackedFile` by `F`. The reason why `ContentFile` is parameterized is that we want to share classes between `DataFile` and `DeleteFile`, like `ManifestReader<F extends ContentFile<F>>`. Then we can return `new ManifestReader<>(...)` to produce both `ManifestReader<DataFile>` and `ManifestReader<DeleteFile>`.
I think that v4 is going to be different because `TrackedFile` is not a parent of `DeleteFile` and `DataFile`. Instead, `TrackedFile` is used as a generic container for `DataFile`, `DeleteFile`, and `ManifestFile` and the v4 root manifest reader can produce all 3 of those types mixed together. The reader will be a `CloseableIterable<TrackedFile>` rather than `CloseableIterable<F extends TrackedFile<F>>` because `TrackedFile` can produce a `DataFile` via `asDataFile()`. It is not always itself a `DataFile`.
To simplify, I think we can drop the type param here and use wrapper classes that call `asDataFile()` or similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540041150
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
+ });
+ }
+
+ private static Function<TrackedFile<?>, TrackedFile<?>> rowIdAssigner(Long firstRowId) {
+ if (firstRowId == null) {
+ return entry -> entry;
Review Comment:
This changes the behavior of row ID assignment in a subtle but incorrect way. When the ID passed in is NULL, the behavior is to explicitly set all `first_row_id` values to NULL. This requirement was added for updating from v2 to v3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-4530720330
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2688515865
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
+ });
+ }
+
+ private static Function<TrackedFile<?>, TrackedFile<?>> rowIdAssigner(Long firstRowId) {
+ if (firstRowId == null) {
+ return entry -> entry;
+ }
+
+ return new Function<>() {
+ private long nextRowId = firstRowId;
+
+ @Override
+ public TrackedFile<?> apply(TrackedFile<?> entry) {
+ if (entry.contentType() == FileContent.DATA) {
Review Comment:
Thanks for that context and it makes sense. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683251726
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
Review Comment:
Moved them to `ManifestStatsStruct`, and made them required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539927310
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
+ }
+
+ @Override
+ public Long pos() {
+ return trackedFile.pos();
+ }
+
+ @Override
+ public int specId() {
+ return trackedFile.partitionSpecId();
+ }
+
+ @Override
+ public FileContent content() {
+ return trackedFile.contentType();
+ }
+
+ @Override
+ public CharSequence path() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public String location() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public FileFormat format() {
+ return trackedFile.fileFormat();
+ }
+
+ @Override
+ public StructLike partition() {
+ return null;
+ }
+
+ @Override
+ public long recordCount() {
+ return trackedFile.recordCount();
+ }
+
+ @Override
+ public long fileSizeInBytes() {
+ Long size = trackedFile.fileSizeInBytes();
+ return size != null ? size : 0L;
+ }
+
+ @Override
+ public Map<Integer, Long> columnSizes() {
+ return null;
Review Comment:
Also TODO. It's important to adapt from the new metadata structure to the old.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539816250
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
Review Comment:
I see. I didn't realize that we had added the setter methods to the `ManifestEntry` interface. Looks like that was probably allowed because `ManifestEntry` is in core and not exposed in the public API. Instead, `DataFile` and `DeleteFile` are in the API.
Here, I think we need to be more careful. This is very likely going to be in API along side `DataFile`, so it should be an immutable interface. To avoid downcasting, the manifest reader should configure its Parquet reader to produce the concrete class, `GenericTrackedFile`. That class can be mutable and then the reader should return instances as `TrackedFile` (because it is `CloseableIterable<TrackedFile>`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539789321
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
Review Comment:
For this API, I think it is cleaner to call `asDataFile()` without any arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539981752
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
+ this(AvroSchemaUtil.convert(avroSchema).asStructType());
+ this.avroSchema = avroSchema;
+ }
+
+ /** Used by internal readers to instantiate this class with a projection schema. */
+ GenericTrackedFile(Types.StructType projection) {
+ super(BASE_TYPE, projection);
+ this.avroSchema = AvroSchemaUtil.convert(projection, "tracked_file");
+ }
+
+ public GenericTrackedFile() {
+ super(BASE_TYPE.fields().size());
+ }
+
+ /**
+ * Full constructor for creating a tracked file.
+ *
+ * @param contentType the type of content (DATA, POSITION_DELETES, etc.)
+ * @param location the file location
+ * @param fileFormat the file format
+ * @param partitionSpecId the partition spec ID
+ * @param recordCount the number of records
+ */
+ public GenericTrackedFile(
+ FileContent contentType,
+ String location,
+ FileFormat fileFormat,
+ int partitionSpecId,
+ long recordCount,
+ Long fileSizeInBytes) {
+ super(BASE_TYPE.fields().size());
+ this.contentType = contentType;
+ this.location = location;
+ this.fileFormat = fileFormat;
+ this.partitionSpecId = partitionSpecId;
+ this.recordCount = recordCount;
+ this.fileSizeInBytes = fileSizeInBytes;
+ }
+
+ /**
+ * Copy constructor.
+ *
+ * @param toCopy a tracked file to copy
+ * @param copyStats whether to copy stats
+ */
+ private GenericTrackedFile(GenericTrackedFile toCopy, boolean copyStats) {
+ super(toCopy);
+
+ // Tracking metadata
+ this.manifestLocation = toCopy.manifestLocation;
+ this.status = toCopy.status;
+ this.snapshotId = toCopy.snapshotId;
+ this.sequenceNumber = toCopy.sequenceNumber;
+ this.fileSequenceNumber = toCopy.fileSequenceNumber;
+ this.firstRowId = toCopy.firstRowId;
+ this.position = toCopy.position;
+
+ // Common file metadata
+ this.contentType = toCopy.contentType;
+ this.location = toCopy.location;
+ this.fileFormat = toCopy.fileFormat;
+ this.recordCount = toCopy.recordCount;
+ this.fileSizeInBytes = toCopy.fileSizeInBytes;
+
+ // Deletion vector
+ this.deletionVectorOffset = toCopy.deletionVectorOffset;
+ this.deletionVectorSizeInBytes = toCopy.deletionVectorSizeInBytes;
+ this.deletionVectorInlineContent =
+ toCopy.deletionVectorInlineContent != null
+ ? Arrays.copyOf(
+ toCopy.deletionVectorInlineContent, toCopy.deletionVectorInlineContent.length)
+ : null;
+
+ // Partition and ordering
+ this.partitionSpecId = toCopy.partitionSpecId;
+ this.sortOrderId = toCopy.sortOrderId;
+
+ // Optional metadata
+ this.keyMetadata =
+ toCopy.keyMetadata != null
+ ? Arrays.copyOf(toCopy.keyMetadata, toCopy.keyMetadata.length)
+ : null;
+ this.splitOffsets =
+ toCopy.splitOffsets != null
+ ? Arrays.copyOf(toCopy.splitOffsets, toCopy.splitOffsets.length)
+ : null;
+ this.equalityIds =
+ toCopy.equalityIds != null
+ ? Arrays.copyOf(toCopy.equalityIds, toCopy.equalityIds.length)
+ : null;
+ this.referencedFile = toCopy.referencedFile;
+
+ if (copyStats) {
+ // Manifest stats
+ this.addedFilesCount = toCopy.addedFilesCount;
+ this.existingFilesCount = toCopy.existingFilesCount;
+ this.deletedFilesCount = toCopy.deletedFilesCount;
+ this.addedRowsCount = toCopy.addedRowsCount;
+ this.existingRowsCount = toCopy.existingRowsCount;
+ this.deletedRowsCount = toCopy.deletedRowsCount;
+ this.minSequenceNumber = toCopy.minSequenceNumber;
+ this.contentStats = toCopy.contentStats;
+ }
+ }
+
+ @Override
+ public String manifestLocation() {
+ return manifestLocation;
+ }
+
+ public void setManifestLocation(String manifestLocation) {
+ this.manifestLocation = manifestLocation;
+ }
+
+ @Override
+ public TrackingInfo trackingInfo() {
Review Comment:
Structs contained within `TrackedFile` should be actual structs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520529287
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
Review Comment:
I'm not sure that we want to pass in the spec, since the record contains an ID. Wouldn't it be better to pass in a map of specs by ID when reading manifests so that this is already known when adapting to `DataFile`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2687996045
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
Review Comment:
That sounds better. Changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683231850
##########
core/src/main/java/org/apache/iceberg/DataTableScan.java:
##########
@@ -63,8 +63,17 @@ protected TableScan newRefinedScan(Table table, Schema schema, TableScanContext
@Override
public CloseableIterable<FileScanTask> doPlanFiles() {
Snapshot snapshot = snapshot();
-
FileIO io = table().io();
+
+ int formatVersion =
+ table() instanceof HasTableOperations
+ ? ((HasTableOperations) table()).operations().current().formatVersion()
+ : 2;
+
+ if (formatVersion >= 4) {
+ return planV4Files(snapshot, io);
Review Comment:
Sounds good. I have reverted this file. Will send a followup PR to wire this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-3747107153
@rdblue Thanks for the thorough feedback! I've updated the PR incorporating pretty much all of it. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-3677067321
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-4473190302
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2687990969
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
Review Comment:
Sounds good. Renamed to `withManifestDV`, following earlier discussion on DVs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539967278
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
Review Comment:
Also, I don't think that this class should be public. Let's keep everything package-private right now.
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
Review Comment:
You can remove this along with the other Avro artifacts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539963991
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
Review Comment:
Each of the sub-structs should be its own struct type (like `ManifestStatsStruct` and `TrackingInfoStruct`) that is read and written the same way that this class is. The `TrackedFile` schema should embed each sub struct as a field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2535173603
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
Review Comment:
Yes, I thought about it, but was not sure which would be cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2535188245
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
Review Comment:
It is in `ManifestEntry.Status`, but it is in `core`. For v4, we need it in the API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539823081
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
Review Comment:
Okay, this is fine for now. We may want to move it later, but we'll see what makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539973390
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
+ this(AvroSchemaUtil.convert(avroSchema).asStructType());
+ this.avroSchema = avroSchema;
+ }
+
+ /** Used by internal readers to instantiate this class with a projection schema. */
+ GenericTrackedFile(Types.StructType projection) {
+ super(BASE_TYPE, projection);
+ this.avroSchema = AvroSchemaUtil.convert(projection, "tracked_file");
+ }
+
+ public GenericTrackedFile() {
+ super(BASE_TYPE.fields().size());
+ }
+
+ /**
+ * Full constructor for creating a tracked file.
+ *
+ * @param contentType the type of content (DATA, POSITION_DELETES, etc.)
+ * @param location the file location
+ * @param fileFormat the file format
+ * @param partitionSpecId the partition spec ID
+ * @param recordCount the number of records
+ */
+ public GenericTrackedFile(
Review Comment:
I don't think that this is a "full constructor". I'm also not sure that we need this constructor at all. It depends on whether we create `TrackedFile` instances directly and I doubt that we will. I think it is more likely that we will use a wrapper on the write path to write `DataFile` as a `TrackedFile` (similar to the `V3Metadata` wrapper classes). In that case, this class only needs to be used in the read path.
Let's remove this constructor for now, unless you need a complete one for test purposes (in which case, it should be package-private).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540011792
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
Review Comment:
The original `ManifestReader` allows callers to project columns by calling `select(String... columns)`, but this isn't a very useful API because the majority of the time that a manifest is being projected, it is by the Iceberg library for a specific purpose. That's why the arguments to `select` are usually `"*"`, `BaseScan.SCAN_WITH_STATS_COLUMNS`, or `BaseScan.SCAN_COLUMNS`. And now that we need to project columnar stats based on fields used in filters, the last two constants are no longer useful.
I think that the v4 implementation should provide more useful projection capabilities:
1. By default, all columns are projected, but without needing to check for `*`
2. A filter-based stats projection that is automatic, based on the filters pushed down to this reader
3. A custom stats projection like `projectStats(Set<Integer> fieldIds)` that sets the content stats projection (to be unioned with the fields from #2)
4. A `project(Schema)` method that can perform more fine-grained projection when the reader is used in metadata table scans and other custom scan situations (like `ManifestFiles.readPaths`).
Removing `select` from the API allows us to simplify the logic in this reader. We don't need to keep lists of fields to pass in or detect when both `select` and `project` are called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540027673
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
Review Comment:
This happens in the old reader because there was no reason to plumb the format through. In v4, this should be passed into the manifest reader instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2687996045
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
Review Comment:
That sounds better. Changing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2688316956
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2529458863
##########
api/src/main/java/org/apache/iceberg/FileContent.java:
##########
@@ -18,11 +18,27 @@
*/
package org.apache.iceberg;
-/** Content type stored in a file, one of DATA, POSITION_DELETES, or EQUALITY_DELETES. */
+/**
+ * Content type stored in a file.
+ *
+ * <p>For V1-V3 tables: DATA, POSITION_DELETES, or EQUALITY_DELETES.
+ *
+ * <p>For V4 tables: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
public enum FileContent {
DATA(0),
POSITION_DELETES(1),
- EQUALITY_DELETES(2);
+ EQUALITY_DELETES(2),
+ /** Data manifest entry (V4+ only) - references data files in a root manifest. */
+ DATA_MANIFEST(3),
+ /** Delete manifest entry (V4+ only) - references delete files in a root manifest. */
+ DELETE_MANIFEST(4),
+ /**
+ * Manifest deletion vector entry (V4+ only) - marks entries in a manifest as deleted without
+ * rewriting the manifest.
+ */
+ MANIFEST_DV(5);
Review Comment:
That was my preference too, and I advocated for it in our community discussion but this is what we settled on. Our current v4 proposal specifically uses `MANIFEST_DV` as a separate content type that references manifests via the referenced_file field. We can certainly change it, but want to hear from others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540075835
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
Review Comment:
Style: In Iceberg, control flow blocks should be separated from the statements following them by an empty newline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540070471
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
Review Comment:
Actually, this is not true. Row ID assignment needs to happen first, then rows can be filtered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539912498
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
+ Types.NestedField SNAPSHOT_ID =
+ Types.NestedField.optional(
+ 1,
+ "snapshot_id",
+ Types.LongType.get(),
+ "Snapshot ID where the file was added, or deleted if status is 2. Inherited when null.");
+ Types.NestedField SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 3,
+ "sequence_number",
+ Types.LongType.get(),
+ "Data sequence number of the file. Inherited when null and status is 1 (added). Must be equal to file_sequence_number if content_type is 3 or 4.");
+ Types.NestedField FILE_SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 4,
+ "file_sequence_number",
+ Types.LongType.get(),
+ "File sequence number indicating when the file was added. Inherited when null and status is added. Must be equal to sequence_number if content_type is 3 or 4.");
+ Types.NestedField FIRST_ROW_ID =
+ Types.NestedField.optional(
+ 142,
+ "first_row_id",
+ Types.LongType.get(),
+ "The _row_id for the first row in the data file if content_type is 0. If content_type is 3, this is the starting _row_id to assign to rows added by ADDED data files.");
+
+ /**
+ * Returns the status of the entry.
+ *
+ * <p>Status values:
+ *
+ * <ul>
+ * <li>0: EXISTING - file was already in the table
+ * <li>1: ADDED - file newly added
+ * <li>2: DELETED - file removed
+ * </ul>
+ */
+ Status status();
+
+ /**
+ * Returns the snapshot ID where the file was added or deleted.
+ *
+ * <p>Inherited when null.
+ */
+ Long snapshotId();
+
+ /**
+ * Returns the data sequence number of the file.
+ *
+ * <p>Inherited when null and status is 1 (added). Must be equal to file_sequence_number if
+ * content_type is 3 or 4.
+ */
+ Long sequenceNumber();
+
+ /**
+ * Returns the file sequence number indicating when the file was added.
+ *
+ * <p>Inherited when null and status is added. Must be equal to sequence_number if content_type is
+ * 3 or 4.
+ */
+ Long fileSequenceNumber();
+
+ /**
+ * Returns the starting row ID for the file.
+ *
+ * <p>If content_type is 0 (DATA), this is the _row_id for the first row in the data file. If
+ * content_type is 3 (DATA_MANIFEST), this is the starting _row_id to assign to rows added by
+ * ADDED data files.
+ */
+ Long firstRowId();
Review Comment:
I think this should also include `manifestLocation()` and `manifestPos()` for tracking information. See https://github.com/apache/iceberg/pull/14533/files#r2539910058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2541106394
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
Review Comment:
What is `content_slice`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539816250
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
Review Comment:
I see. I didn't realize that we had added the setter methods to the `ManifestEntry` interface. Looks like that was probably allowed because `ManifestEntry` is in core and not exposed in the public API. Instead, `DataFile` and `DeleteFile` are in the API.
Here, I think we need to be more careful. This is very likely going to be in API along side `DataFile`, so it should be an immutable interface. To avoid downcasting, the manifest reader should configure its Parquet reader to produce the concrete class, `GenericTrackedFile`. That class can be mutable so the reader can pass instances to `InheritableTrackedMetadata` and then the reader should return those instances as `TrackedFile` (because it is `CloseableIterable<TrackedFile>`). That shouldn't require casting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520547182
##########
core/src/main/java/org/apache/iceberg/DataTableScan.java:
##########
@@ -63,8 +63,17 @@ protected TableScan newRefinedScan(Table table, Schema schema, TableScanContext
@Override
public CloseableIterable<FileScanTask> doPlanFiles() {
Snapshot snapshot = snapshot();
-
FileIO io = table().io();
+
+ int formatVersion =
+ table() instanceof HasTableOperations
+ ? ((HasTableOperations) table()).operations().current().formatVersion()
+ : 2;
+
+ if (formatVersion >= 4) {
+ return planV4Files(snapshot, io);
Review Comment:
Rather than modifying data table scan right now, let's leave this out. We don't need to plug anything into table scans at this point, since that is just a configuration API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14533: [WIP] V4 Manifest Read Support
URL: https://github.com/apache/iceberg/pull/14533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539865778
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadata.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+
+/**
+ * Interface for applying inheritable metadata to tracked file entries in V4 manifests.
+ *
+ * <p>Similar to {@link InheritableMetadata} but for V4 TrackedFile entries.
+ */
+interface InheritableTrackedMetadata extends Serializable {
+ /**
+ * Apply inheritable metadata to a tracked file entry.
+ *
+ * @param entry the tracked file entry
+ * @return the entry with metadata applied
+ */
+ TrackedFile<?> apply(TrackedFile<?> entry);
Review Comment:
This should either be `GenericTrackedFile` (the internal representation) or an interface in core, like `TrackedFileSetters`. I would probably use `GenericTrackedFile`.
I think we also need to consider what setter methods to add here. We have inverted the relationship between `TrackedFile` / `DataFile` and `TrackingInfo` / `ManifestEntry` so I don't think it makes sense to call `setSnapshotId` on `TrackedFile` itself just because that is the top-level record in manifests. I'd rather get the `TrackingInfo` and modify that directly:
```java
TrackedFile apply(GenericTrackedFile file) {
TrackingInfoStruct info;
if (file.trackingInfo() instanceof TrackingInfoStruct) {
info = file.trackingInfo();
} else {
info = new TrackingInfoStruct(file.trackingInfo()); // copy to mutable
}
if (info.snapshotId() == null) {
info.setSnapshotId(snapshotId);
}
...
file.setTrackingInfo(info);
return file;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539917506
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
Review Comment:
I would rather have this check in `asDataFile` to avoid allocation before running the check. I'd also probably make this a private inner class of `TrackedFile` to avoid exposing it and needing to lock down the constructor and check the content like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540073920
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
Review Comment:
I think that it is a bad practice to assume a row should be filtered (or some other action) based on a missing value like this, similar to the missing `tracking_info` cases below.
The problem is that this will cause *silent data loss* if `pos` is incorrectly projected as null because of a bug in another part of the codebase. This filter depends on having a valid `pos` so it should fail if `pos` is null rather than choosing to skip data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2535178838
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
Review Comment:
I had it as immutable initially, and added mutability while implementing inheritable metadata. That means that in places which need mutability, we will need to downcast it, which is fine I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539839538
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
+ Types.NestedField SNAPSHOT_ID =
+ Types.NestedField.optional(
+ 1,
+ "snapshot_id",
+ Types.LongType.get(),
+ "Snapshot ID where the file was added, or deleted if status is 2. Inherited when null.");
+ Types.NestedField SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 3,
+ "sequence_number",
+ Types.LongType.get(),
+ "Data sequence number of the file. Inherited when null and status is 1 (added). Must be equal to file_sequence_number if content_type is 3 or 4.");
Review Comment:
Similar to my comment above, this should not describe behavior. Instead, describe what the data sequence number means.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539960982
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
Review Comment:
The prefix `Generic` was used to identify Avro generic classes that are compatible with `IndexedRecord`. Now that we have replaced `IndexedRecord` with `StructLike`, I'd recommend renaming this and other classes to `TrackedFileStruct`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540016469
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
Review Comment:
This should still accept `specsById` to set the spec for each data or delete file. We don't want to require callers to do something like this that requires the caller to correctly find a spec:
```java
trackedFile.asDataFile(specsById.get(trackedFile.specId()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539816250
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
Review Comment:
I see. I didn't realize that we had added the setter methods to the `ManifestEntry` interface. Looks like that was probably allowed because `ManifestEntry` is in core and not exposed in the public API. Instead, `DataFile` and `DeleteFile` are in the API.
Here, I think we need to be more careful. This is very likely going to be in API along side `DataFile`, so it should be an immutable interface. To avoid downcasting, the manifest reader should configure its Parquet reader to produce the concrete class, `GenericTrackedFile`. That class can be mutable so the reader can pass instances to `InheritableTrackedMetadata` and then the reader should return those instances as `TrackedFile` (because it is `CloseableIterable<TrackedFile>`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683234247
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520535234
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
Review Comment:
This API should not expose any setter methods. The implementation used can, if needed, for things like inherited metadata. But the API interface itself should not force implementations to be mutable. In general, we want to think of the API interfaces as immutable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520505191
##########
api/src/main/java/org/apache/iceberg/FileContent.java:
##########
@@ -18,11 +18,27 @@
*/
package org.apache.iceberg;
-/** Content type stored in a file, one of DATA, POSITION_DELETES, or EQUALITY_DELETES. */
+/**
+ * Content type stored in a file.
+ *
+ * <p>For V1-V3 tables: DATA, POSITION_DELETES, or EQUALITY_DELETES.
+ *
+ * <p>For V4 tables: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
public enum FileContent {
DATA(0),
POSITION_DELETES(1),
- EQUALITY_DELETES(2);
+ EQUALITY_DELETES(2),
+ /** Data manifest entry (V4+ only) - references data files in a root manifest. */
+ DATA_MANIFEST(3),
+ /** Delete manifest entry (V4+ only) - references delete files in a root manifest. */
+ DELETE_MANIFEST(4),
+ /**
+ * Manifest deletion vector entry (V4+ only) - marks entries in a manifest as deleted without
+ * rewriting the manifest.
+ */
+ MANIFEST_DV(5);
Review Comment:
I prefer the option of having the DV located in a field of the data or delete manifest record. That way we don't have to wait to find the DV before processing a manifest file. Not sure what others think here, but since the DV metadata/content is likely going to be different between the Metadata DV (inline) and Data DV (stored in Puffin), I don't see much value in trying to reuse metadata fields for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539842120
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
+ Types.NestedField SNAPSHOT_ID =
+ Types.NestedField.optional(
+ 1,
+ "snapshot_id",
+ Types.LongType.get(),
+ "Snapshot ID where the file was added, or deleted if status is 2. Inherited when null.");
+ Types.NestedField SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 3,
+ "sequence_number",
+ Types.LongType.get(),
+ "Data sequence number of the file. Inherited when null and status is 1 (added). Must be equal to file_sequence_number if content_type is 3 or 4.");
+ Types.NestedField FILE_SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 4,
+ "file_sequence_number",
+ Types.LongType.get(),
+ "File sequence number indicating when the file was added. Inherited when null and status is added. Must be equal to sequence_number if content_type is 3 or 4.");
+ Types.NestedField FIRST_ROW_ID =
+ Types.NestedField.optional(
+ 142,
+ "first_row_id",
+ Types.LongType.get(),
+ "The _row_id for the first row in the data file if content_type is 0. If content_type is 3, this is the starting _row_id to assign to rows added by ADDED data files.");
+
+ /**
+ * Returns the status of the entry.
+ *
+ * <p>Status values:
+ *
+ * <ul>
+ * <li>0: EXISTING - file was already in the table
+ * <li>1: ADDED - file newly added
+ * <li>2: DELETED - file removed
+ * </ul>
+ */
+ Status status();
+
+ /**
+ * Returns the snapshot ID where the file was added or deleted.
+ *
+ * <p>Inherited when null.
+ */
+ Long snapshotId();
+
+ /**
+ * Returns the data sequence number of the file.
+ *
+ * <p>Inherited when null and status is 1 (added). Must be equal to file_sequence_number if
+ * content_type is 3 or 4.
+ */
+ Long sequenceNumber();
+
+ /**
+ * Returns the file sequence number indicating when the file was added.
+ *
+ * <p>Inherited when null and status is added. Must be equal to sequence_number if content_type is
+ * 3 or 4.
+ */
+ Long fileSequenceNumber();
+
+ /**
+ * Returns the starting row ID for the file.
+ *
+ * <p>If content_type is 0 (DATA), this is the _row_id for the first row in the data file. If
+ * content_type is 3 (DATA_MANIFEST), this is the starting _row_id to assign to rows added by
+ * ADDED data files.
Review Comment:
As with the schema fields, I don't think that it is valuable to describe the format behavior here, either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539920436
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
Review Comment:
Using an inner class would allow you to reference the private variables directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539914088
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
Review Comment:
What about `TrackedDataFile`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539899636
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadataFactory.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/** Factory for creating {@link InheritableTrackedMetadata} instances. */
+class InheritableTrackedMetadataFactory {
+
+ private InheritableTrackedMetadataFactory() {}
+
+ /**
+ * Creates inheritable metadata from explicit values.
+ *
+ * @param snapshotId the snapshot ID
+ * @param sequenceNumber the sequence number
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata create(long snapshotId, long sequenceNumber) {
+ return new BaseInheritableTrackedMetadata(snapshotId, sequenceNumber);
+ }
+
+ /**
+ * Creates inheritable metadata from a tracked file representing a manifest (for reading leaf
+ * manifests).
+ *
+ * @param manifestEntry the DATA_MANIFEST or DELETE_MANIFEST tracked file from root
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata fromTrackedFile(TrackedFile<?> manifestEntry) {
+ Preconditions.checkArgument(
+ manifestEntry.contentType() == FileContent.DATA_MANIFEST
+ || manifestEntry.contentType() == FileContent.DELETE_MANIFEST,
+ "Can only create metadata from manifest entries, got: %s",
+ manifestEntry.contentType());
+
+ TrackingInfo tracking = manifestEntry.trackingInfo();
+ Long snapshotId = tracking != null ? tracking.snapshotId() : null;
Review Comment:
Tracking info should never be null for a manifest passed into `fromManifest`. If it is, then this should complain that the manifest is missing its tracking info and appears to be uncommitted. (For context, to read an uncommitted manifest, the caller must use either `InheritableTrackingMetadata.forCopy` or `InheritableTrackingMetadata.empty()` but we _don't_ want to include this hint in the error message.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-3677099047
Thanks for the feedback, Ryan. I will send a new revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683393130
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
Review Comment:
Sounds better - renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2688043640
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
Review Comment:
My preference is for `V4ManifestReader` to stay simple - it just returns `TrackedFile` instances. The spec lookup and conversion to `DataFile`/`DeleteFile` should happen in a higher-level component like `ManifestExpander` (which already has specsById). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2688311158
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
Review Comment:
Good catch. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539825598
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
Review Comment:
I'd like to have a static `schema()` method on these classes that defines the current schema, not just fields. That way it's easy to see how the schema is constructed and easy to reference for the read path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539834543
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
Review Comment:
Other field docs expand the enum symbols: `"Entry status: 0=existing, 1=added, 2=removed"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539975742
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
+ this(AvroSchemaUtil.convert(avroSchema).asStructType());
+ this.avroSchema = avroSchema;
+ }
+
+ /** Used by internal readers to instantiate this class with a projection schema. */
+ GenericTrackedFile(Types.StructType projection) {
+ super(BASE_TYPE, projection);
+ this.avroSchema = AvroSchemaUtil.convert(projection, "tracked_file");
+ }
+
+ public GenericTrackedFile() {
+ super(BASE_TYPE.fields().size());
+ }
+
+ /**
+ * Full constructor for creating a tracked file.
+ *
+ * @param contentType the type of content (DATA, POSITION_DELETES, etc.)
+ * @param location the file location
+ * @param fileFormat the file format
+ * @param partitionSpecId the partition spec ID
+ * @param recordCount the number of records
+ */
+ public GenericTrackedFile(
+ FileContent contentType,
+ String location,
+ FileFormat fileFormat,
+ int partitionSpecId,
+ long recordCount,
+ Long fileSizeInBytes) {
+ super(BASE_TYPE.fields().size());
+ this.contentType = contentType;
+ this.location = location;
+ this.fileFormat = fileFormat;
+ this.partitionSpecId = partitionSpecId;
+ this.recordCount = recordCount;
+ this.fileSizeInBytes = fileSizeInBytes;
+ }
+
+ /**
+ * Copy constructor.
+ *
+ * @param toCopy a tracked file to copy
+ * @param copyStats whether to copy stats
+ */
+ private GenericTrackedFile(GenericTrackedFile toCopy, boolean copyStats) {
Review Comment:
For v4, we can remove the paths that use a `copyStats` boolean. Instead, we should use a set of field IDs to identify the fields to copy, much like the newer copy API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539891816
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadataFactory.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/** Factory for creating {@link InheritableTrackedMetadata} instances. */
+class InheritableTrackedMetadataFactory {
+
+ private InheritableTrackedMetadataFactory() {}
+
+ /**
+ * Creates inheritable metadata from explicit values.
+ *
+ * @param snapshotId the snapshot ID
+ * @param sequenceNumber the sequence number
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata create(long snapshotId, long sequenceNumber) {
+ return new BaseInheritableTrackedMetadata(snapshotId, sequenceNumber);
+ }
+
+ /**
+ * Creates inheritable metadata from a tracked file representing a manifest (for reading leaf
+ * manifests).
+ *
+ * @param manifestEntry the DATA_MANIFEST or DELETE_MANIFEST tracked file from root
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata fromTrackedFile(TrackedFile<?> manifestEntry) {
Review Comment:
Methods like this should accept a typed file, like `ManifestFile`, instead of converting. That way we can keep the conversion logic in a single place and the caller can either pass an existing `ManifestFile` or calls `trackedFile.asManifestFile()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539785949
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
Review Comment:
Let's just assign it now and update the proposal. I also have example assignments in my exploration if you want to be consistent:
```rust
fn tracked_file_write_schema(
table_schema: &Schema,
stats_modes: Option<&HashMap<FieldId, StatsMode>>,
) -> Schema {
let default = HashMap::from_iter((0..=32).into_iter().map(|id| (id, DEFAULT_STATS_MODE)));
let modes: &HashMap<FieldId, StatsMode> = stats_modes.unwrap_or_else(|| &default);
StructType::new([
required("tracking_info", tracking_info_schema(), 147), // TODO: assigned ID
required("content", DataType::INTEGER, 134), // now required!
required("location", DataType::STRING, 100),
required("file_format", DataType::STRING, 101),
required("record_count", DataType::LONG, 103),
required("file_size_in_bytes", DataType::LONG, 104),
required(
"content_stats",
content_stats_schema(table_schema, modes),
146,
), // TODO: ID is from stats proposal
optional("key_metadata", DataType::BINARY, 131),
optional("split_offsets", ArrayType::new(DataType::LONG, false), 132), // TODO: missing element ID (133)
optional("content_slice", slice_schema(), 148), // TODO: assigned ID
optional("referenced_file", DataType::STRING, 143),
optional("manifest_stats", manifest_stats_schema(), 149), // TODO: assigned ID
optional("min_sequence_number", DataType::LONG, 516),
])
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540058960
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
+ });
+ }
+
+ private static Function<TrackedFile<?>, TrackedFile<?>> rowIdAssigner(Long firstRowId) {
+ if (firstRowId == null) {
+ return entry -> entry;
+ }
+
+ return new Function<>() {
+ private long nextRowId = firstRowId;
+
+ @Override
+ public TrackedFile<?> apply(TrackedFile<?> entry) {
+ if (entry.contentType() == FileContent.DATA) {
Review Comment:
I think the logic here is mostly correct, but can be improved. I agree that `first_row_id` is only assigned if the file is a data file. It should also only be assigned in v4 if this is a leaf manifest -- so we probably don't even want to run an ID assigner on the root manifest because the root be written with explicit `first_row_id` values, just like the manifest list is in v3.
The main issue that I see here is that this only assigns `first_row_id` if the tracking info is non-null. There should be no case where tracking info is null. For any file, the only time tracking info should be null is when it is not yet written to a manifest or is to be written to a manifest (and so tracking info will be supressed/overwritten). There's no case where it should be null in this class because this is reading from a manifest.
The only exception to that (the paragraph just above) is when reading with a custom projection schema -- meaning that the file is being read with a projection through a metadata table. For example:
```sql
SELECT location, tracking_info.first_row_id FROM db.table.data_files
```
In cases like this, we need to ensure that the necessary information to produce `first_row_id` is present. In the current reader, we _always_ project `first_row_id` and I would suggest always tracking the entirety of `tracking_info` so that we know problems from the wrong `first_row_id` will not be possible. We can always project more fields than requested, so I'd just make sure all fields of `tracking_info` are projected any time this reader is used.
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
Review Comment:
Prefer the built-in `UncheckedIOException` to `RuntimeIOException`.
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
Review Comment:
Use `ByteBuffers.toByteArray` instead of copying directly.
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
+ });
+ }
+
+ private static Function<TrackedFile<?>, TrackedFile<?>> rowIdAssigner(Long firstRowId) {
+ if (firstRowId == null) {
+ return entry -> entry;
+ }
+
+ return new Function<>() {
+ private long nextRowId = firstRowId;
+
+ @Override
+ public TrackedFile<?> apply(TrackedFile<?> entry) {
+ if (entry.contentType() == FileContent.DATA) {
Review Comment:
I think the logic here is mostly correct, but can be improved. I agree that `first_row_id` is only assigned if the file is a data file. It should also only be assigned in v4 if this is a leaf manifest -- so we probably don't even want to run an ID assigner on the root manifest because the root be written with explicit `first_row_id` values, just like the manifest list is in v3.
The main issue that I see here is that this only assigns `first_row_id` if the tracking info is non-null. There should be no case where tracking info is null. For any file, the only time tracking info should be null is when it is not yet written to a manifest or is to be written to a manifest (and so tracking info will be supressed/overwritten). There's no case where it should be null in this class because this is reading from a manifest.
The only exception to that (the paragraph just above) is when reading with a custom projection schema -- meaning that the file is being read with a projection through a metadata table. For example:
```sql
SELECT location, tracking_info.first_row_id FROM db.table.data_files
```
In cases like this, we need to ensure that the necessary information to produce `first_row_id` is present. In the current reader, we _always_ project `first_row_id` and I would suggest always tracking the entirety of `tracking_info` so that we know problems from the wrong `first_row_id` will not be possible. We can always project more fields than requested, so I'd just make sure all fields of `tracking_info` are projected any time this reader is used.
If `tracking_info` is assumed to always be present, then this should not check whether it is here. Instead, this should assume it is non-null and throw an NPE if that assumption is violated.
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
Review Comment:
This is fine for now, but if the DV is tracked by the same entry as the manifest itself, then this should be passed in through the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2678075893
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
Review Comment:
I assume this is `content_info` struct as in the proposal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683271870
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
Review Comment:
Done. BTW is `SupportsIndexProjection` still useful? I kept it for now, since I wasn't sure if we needed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683383606
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadataFactory.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/** Factory for creating {@link InheritableTrackedMetadata} instances. */
+class InheritableTrackedMetadataFactory {
+
+ private InheritableTrackedMetadataFactory() {}
+
+ /**
+ * Creates inheritable metadata from explicit values.
+ *
+ * @param snapshotId the snapshot ID
+ * @param sequenceNumber the sequence number
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata create(long snapshotId, long sequenceNumber) {
+ return new BaseInheritableTrackedMetadata(snapshotId, sequenceNumber);
+ }
+
+ /**
+ * Creates inheritable metadata from a tracked file representing a manifest (for reading leaf
+ * manifests).
+ *
+ * @param manifestEntry the DATA_MANIFEST or DELETE_MANIFEST tracked file from root
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata fromTrackedFile(TrackedFile<?> manifestEntry) {
Review Comment:
Removed it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-4270792361
still active
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2535199473
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
Review Comment:
It will be supported. I left it out for now because the stats was still being defined by @nastra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2523906797
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
Review Comment:
Is it intentional that we removed `copyWithStats(Set<Integer> requestedColumnIds)` from `ContentFile`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2529459895
##########
api/src/main/java/org/apache/iceberg/DeletionVector.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Deletion vector information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This encapsulates both inline and out-of-line deletion vector content. This struct must be
+ * defined for types 1 (POSITION_DELETES) and 5 (MANIFEST_DV).
+ */
+public interface DeletionVector {
+ Types.NestedField OFFSET =
+ Types.NestedField.optional(
+ 144, "offset", Types.LongType.get(), "Offset in the file where the content starts");
+ Types.NestedField SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 145,
+ "size_in_bytes",
+ Types.LongType.get(),
+ "Length of a referenced content stored in the file; required if offset is present");
+ Types.NestedField INLINE_CONTENT =
+ Types.NestedField.optional(
+ 146, "inline_content", Types.BinaryType.get(), "Serialized bitmap for inline DVs");
+
+ /**
+ * Returns the offset in the file where the deletion vector content starts.
+ *
+ * <p>This is used for out-of-line deletion vectors stored in Puffin files.
+ */
+ Long offset();
+
+ /**
+ * Returns the size in bytes of the deletion vector content.
+ *
+ * <p>Required if offset is present.
+ */
+ Long sizeInBytes();
+
+ /**
+ * Returns the serialized bitmap for inline deletion vectors.
+ *
+ * <p>When present, the deletion vector is stored inline in the manifest rather than in a separate
+ * Puffin file.
+ */
+ ByteBuffer inlineContent();
Review Comment:
Data and metadata DVs seemed like same concepts to me. I can create a separate `ManifestDeletionVector`.
##########
api/src/main/java/org/apache/iceberg/DeletionVector.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Deletion vector information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This encapsulates both inline and out-of-line deletion vector content. This struct must be
+ * defined for types 1 (POSITION_DELETES) and 5 (MANIFEST_DV).
+ */
+public interface DeletionVector {
+ Types.NestedField OFFSET =
+ Types.NestedField.optional(
+ 144, "offset", Types.LongType.get(), "Offset in the file where the content starts");
+ Types.NestedField SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 145,
+ "size_in_bytes",
+ Types.LongType.get(),
+ "Length of a referenced content stored in the file; required if offset is present");
+ Types.NestedField INLINE_CONTENT =
+ Types.NestedField.optional(
+ 146, "inline_content", Types.BinaryType.get(), "Serialized bitmap for inline DVs");
+
+ /**
+ * Returns the offset in the file where the deletion vector content starts.
+ *
+ * <p>This is used for out-of-line deletion vectors stored in Puffin files.
+ */
+ Long offset();
+
+ /**
+ * Returns the size in bytes of the deletion vector content.
+ *
+ * <p>Required if offset is present.
+ */
+ Long sizeInBytes();
+
+ /**
+ * Returns the serialized bitmap for inline deletion vectors.
+ *
+ * <p>When present, the deletion vector is stored inline in the manifest rather than in a separate
+ * Puffin file.
+ */
+ ByteBuffer inlineContent();
Review Comment:
Data and metadata DVs seemed like similar concepts to me. I can create a separate `ManifestDeletionVector`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2535171734
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
Review Comment:
Yep. Will add it as soon as we define it. Right now it's TBD in the proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539984431
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
+ this(AvroSchemaUtil.convert(avroSchema).asStructType());
+ this.avroSchema = avroSchema;
+ }
+
+ /** Used by internal readers to instantiate this class with a projection schema. */
+ GenericTrackedFile(Types.StructType projection) {
+ super(BASE_TYPE, projection);
+ this.avroSchema = AvroSchemaUtil.convert(projection, "tracked_file");
+ }
+
+ public GenericTrackedFile() {
+ super(BASE_TYPE.fields().size());
+ }
+
+ /**
+ * Full constructor for creating a tracked file.
+ *
+ * @param contentType the type of content (DATA, POSITION_DELETES, etc.)
+ * @param location the file location
+ * @param fileFormat the file format
+ * @param partitionSpecId the partition spec ID
+ * @param recordCount the number of records
+ */
+ public GenericTrackedFile(
+ FileContent contentType,
+ String location,
+ FileFormat fileFormat,
+ int partitionSpecId,
+ long recordCount,
+ Long fileSizeInBytes) {
+ super(BASE_TYPE.fields().size());
+ this.contentType = contentType;
+ this.location = location;
+ this.fileFormat = fileFormat;
+ this.partitionSpecId = partitionSpecId;
+ this.recordCount = recordCount;
+ this.fileSizeInBytes = fileSizeInBytes;
+ }
+
+ /**
+ * Copy constructor.
+ *
+ * @param toCopy a tracked file to copy
+ * @param copyStats whether to copy stats
+ */
+ private GenericTrackedFile(GenericTrackedFile toCopy, boolean copyStats) {
+ super(toCopy);
+
+ // Tracking metadata
+ this.manifestLocation = toCopy.manifestLocation;
+ this.status = toCopy.status;
+ this.snapshotId = toCopy.snapshotId;
+ this.sequenceNumber = toCopy.sequenceNumber;
+ this.fileSequenceNumber = toCopy.fileSequenceNumber;
+ this.firstRowId = toCopy.firstRowId;
+ this.position = toCopy.position;
+
+ // Common file metadata
+ this.contentType = toCopy.contentType;
+ this.location = toCopy.location;
+ this.fileFormat = toCopy.fileFormat;
+ this.recordCount = toCopy.recordCount;
+ this.fileSizeInBytes = toCopy.fileSizeInBytes;
+
+ // Deletion vector
+ this.deletionVectorOffset = toCopy.deletionVectorOffset;
+ this.deletionVectorSizeInBytes = toCopy.deletionVectorSizeInBytes;
+ this.deletionVectorInlineContent =
+ toCopy.deletionVectorInlineContent != null
+ ? Arrays.copyOf(
+ toCopy.deletionVectorInlineContent, toCopy.deletionVectorInlineContent.length)
+ : null;
+
+ // Partition and ordering
+ this.partitionSpecId = toCopy.partitionSpecId;
+ this.sortOrderId = toCopy.sortOrderId;
+
+ // Optional metadata
+ this.keyMetadata =
+ toCopy.keyMetadata != null
+ ? Arrays.copyOf(toCopy.keyMetadata, toCopy.keyMetadata.length)
+ : null;
+ this.splitOffsets =
+ toCopy.splitOffsets != null
+ ? Arrays.copyOf(toCopy.splitOffsets, toCopy.splitOffsets.length)
+ : null;
+ this.equalityIds =
+ toCopy.equalityIds != null
+ ? Arrays.copyOf(toCopy.equalityIds, toCopy.equalityIds.length)
+ : null;
+ this.referencedFile = toCopy.referencedFile;
+
+ if (copyStats) {
+ // Manifest stats
+ this.addedFilesCount = toCopy.addedFilesCount;
+ this.existingFilesCount = toCopy.existingFilesCount;
+ this.deletedFilesCount = toCopy.deletedFilesCount;
+ this.addedRowsCount = toCopy.addedRowsCount;
+ this.existingRowsCount = toCopy.existingRowsCount;
+ this.deletedRowsCount = toCopy.deletedRowsCount;
+ this.minSequenceNumber = toCopy.minSequenceNumber;
+ this.contentStats = toCopy.contentStats;
+ }
+ }
+
+ @Override
+ public String manifestLocation() {
+ return manifestLocation;
+ }
+
+ public void setManifestLocation(String manifestLocation) {
+ this.manifestLocation = manifestLocation;
+ }
+
+ @Override
+ public TrackingInfo trackingInfo() {
+ if (status == null
+ && snapshotId == null
+ && sequenceNumber == null
+ && fileSequenceNumber == null
+ && firstRowId == null) {
+ return null;
+ }
+
+ return new TrackingInfo() {
+ @Override
+ public TrackingInfo.Status status() {
+ return status;
+ }
+
+ @Override
+ public Long snapshotId() {
+ return snapshotId;
+ }
+
+ @Override
+ public Long sequenceNumber() {
+ return sequenceNumber;
+ }
+
+ @Override
+ public Long fileSequenceNumber() {
+ return fileSequenceNumber;
+ }
+
+ @Override
+ public Long firstRowId() {
+ return firstRowId;
+ }
+ };
+ }
+
+ @Override
+ public Long pos() {
+ return position;
+ }
+
+ @Override
+ public void setStatus(TrackingInfo.Status status) {
+ this.status = status;
+ }
+
+ @Override
+ public void setSnapshotId(Long snapshotId) {
+ this.snapshotId = snapshotId;
+ }
+
+ @Override
+ public void setSequenceNumber(Long sequenceNumber) {
+ this.sequenceNumber = sequenceNumber;
+ }
+
+ @Override
+ public void setFileSequenceNumber(Long fileSequenceNumber) {
+ this.fileSequenceNumber = fileSequenceNumber;
+ }
+
+ @Override
+ public void setFirstRowId(Long firstRowId) {
+ this.firstRowId = firstRowId;
+ }
+
+ @Override
+ public void setPos(Long pos) {
+ this.position = pos;
+ }
+
+ @Override
+ public FileContent contentType() {
+ return contentType;
+ }
+
+ public void setContentType(FileContent contentType) {
+ this.contentType = contentType;
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ public void setLocation(String location) {
+ this.location = location;
+ }
+
+ @Override
+ public FileFormat fileFormat() {
+ return fileFormat;
+ }
+
+ public void setFileFormat(FileFormat fileFormat) {
+ this.fileFormat = fileFormat;
+ }
+
+ @Override
+ public DeletionVector deletionVector() {
+ if (deletionVectorOffset == null
+ && deletionVectorSizeInBytes == null
+ && deletionVectorInlineContent == null) {
+ return null;
+ }
+
+ return new DeletionVector() {
+ @Override
+ public Long offset() {
+ return deletionVectorOffset;
+ }
+
+ @Override
+ public Long sizeInBytes() {
+ return deletionVectorSizeInBytes;
+ }
+
+ @Override
+ public ByteBuffer inlineContent() {
+ return deletionVectorInlineContent != null
+ ? ByteBuffer.wrap(deletionVectorInlineContent)
+ : null;
+ }
+ };
+ }
+
+ public void setDeletionVectorOffset(Long offset) {
+ this.deletionVectorOffset = offset;
+ }
+
+ public void setDeletionVectorSizeInBytes(Long sizeInBytes) {
+ this.deletionVectorSizeInBytes = sizeInBytes;
+ }
+
+ public void setDeletionVectorInlineContent(ByteBuffer inlineContent) {
+ this.deletionVectorInlineContent = ByteBuffers.toByteArray(inlineContent);
+ }
+
+ @Override
+ public int partitionSpecId() {
+ return partitionSpecId;
+ }
+
+ public void setPartitionSpecId(int partitionSpecId) {
+ this.partitionSpecId = partitionSpecId;
+ }
+
+ @Override
+ public Integer sortOrderId() {
+ return sortOrderId;
+ }
+
+ public void setSortOrderId(Integer sortOrderId) {
+ this.sortOrderId = sortOrderId;
+ }
+
+ @Override
+ public long recordCount() {
+ return recordCount;
+ }
+
+ public void setRecordCount(long recordCount) {
+ this.recordCount = recordCount;
+ }
+
+ @Override
+ public Long fileSizeInBytes() {
+ return fileSizeInBytes;
+ }
+
+ public void setFileSizeInBytes(Long fileSizeInBytes) {
+ this.fileSizeInBytes = fileSizeInBytes;
+ }
+
+ @Override
+ public Object contentStats() {
+ return contentStats;
+ }
+
+ @Override
+ public ManifestStats manifestStats() {
+ if (addedFilesCount == null
+ && existingFilesCount == null
+ && deletedFilesCount == null
+ && addedRowsCount == null
+ && existingRowsCount == null
+ && deletedRowsCount == null
+ && minSequenceNumber == null) {
+ return null;
+ }
+
+ return new ManifestStats() {
+ @Override
+ public int addedFilesCount() {
+ return addedFilesCount != null ? addedFilesCount : 0;
+ }
+
+ @Override
+ public int existingFilesCount() {
+ return existingFilesCount != null ? existingFilesCount : 0;
+ }
+
+ @Override
+ public int deletedFilesCount() {
+ return deletedFilesCount != null ? deletedFilesCount : 0;
+ }
+
+ @Override
+ public long addedRowsCount() {
+ return addedRowsCount != null ? addedRowsCount : 0L;
+ }
+
+ @Override
+ public long existingRowsCount() {
+ return existingRowsCount != null ? existingRowsCount : 0L;
+ }
+
+ @Override
+ public long deletedRowsCount() {
+ return deletedRowsCount != null ? deletedRowsCount : 0L;
+ }
+
+ @Override
+ public long minSequenceNumber() {
+ return minSequenceNumber != null ? minSequenceNumber : 0L;
+ }
+ };
+ }
+
+ public void setAddedFilesCount(Integer count) {
+ this.addedFilesCount = count;
+ }
+
+ public void setExistingFilesCount(Integer count) {
+ this.existingFilesCount = count;
+ }
+
+ public void setDeletedFilesCount(Integer count) {
+ this.deletedFilesCount = count;
+ }
+
+ public void setAddedRowsCount(Long count) {
+ this.addedRowsCount = count;
+ }
+
+ public void setExistingRowsCount(Long count) {
+ this.existingRowsCount = count;
+ }
+
+ public void setDeletedRowsCount(Long count) {
+ this.deletedRowsCount = count;
+ }
+
+ public void setMinSequenceNumber(Long minSeqNum) {
+ this.minSequenceNumber = minSeqNum;
+ }
+
+ @Override
+ public ByteBuffer keyMetadata() {
+ return keyMetadata != null ? ByteBuffer.wrap(keyMetadata) : null;
+ }
+
+ public void setKeyMetadata(ByteBuffer keyMetadata) {
+ this.keyMetadata = ByteBuffers.toByteArray(keyMetadata);
+ }
+
+ @Override
+ public List<Long> splitOffsets() {
+ return ArrayUtil.toUnmodifiableLongList(splitOffsets);
+ }
+
+ public void setSplitOffsets(List<Long> offsets) {
+ this.splitOffsets = ArrayUtil.toLongArray(offsets);
+ }
+
+ @Override
+ public List<Integer> equalityIds() {
+ return ArrayUtil.toUnmodifiableIntList(equalityIds);
+ }
+
+ public void setEqualityIds(List<Integer> ids) {
+ this.equalityIds = ArrayUtil.toIntArray(ids);
+ }
+
+ @Override
+ public String referencedFile() {
+ return referencedFile;
+ }
+
+ public void setReferencedFile(String referencedFile) {
+ this.referencedFile = referencedFile;
+ }
+
+ @Override
+ public GenericTrackedFile copy() {
+ return new GenericTrackedFile(this, true);
+ }
+
+ @Override
+ public GenericTrackedFile copyWithoutStats() {
+ return new GenericTrackedFile(this, false);
+ }
+
+ @Override
+ public GenericTrackedFile copyWithStats(java.util.Set<Integer> requestedColumnIds) {
+ // TODO: When ContentStats structure is implemented, filter stats to only requestedColumnIds
+ // For now, return a full copy with all stats since contentStats is just Object
+ return new GenericTrackedFile(this, true);
+ }
+
+ @Override
+ public Schema getSchema() {
+ if (avroSchema == null) {
+ this.avroSchema = getAvroSchema();
+ }
+ return avroSchema;
+ }
+
+ @Override
+ public void put(int i, Object v) {
Review Comment:
Not needed, same as `getSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539977913
##########
api/src/main/java/org/apache/iceberg/FileContent.java:
##########
@@ -18,11 +18,27 @@
*/
package org.apache.iceberg;
-/** Content type stored in a file, one of DATA, POSITION_DELETES, or EQUALITY_DELETES. */
+/**
+ * Content type stored in a file.
+ *
+ * <p>For V1-V3 tables: DATA, POSITION_DELETES, or EQUALITY_DELETES.
+ *
+ * <p>For V4 tables: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
public enum FileContent {
DATA(0),
POSITION_DELETES(1),
- EQUALITY_DELETES(2);
+ EQUALITY_DELETES(2),
+ /** Data manifest entry (V4+ only) - references data files in a root manifest. */
+ DATA_MANIFEST(3),
+ /** Delete manifest entry (V4+ only) - references delete files in a root manifest. */
+ DELETE_MANIFEST(4),
+ /**
+ * Manifest deletion vector entry (V4+ only) - marks entries in a manifest as deleted without
+ * rewriting the manifest.
+ */
+ MANIFEST_DV(5);
Review Comment:
I just check with Amogh and I don't think that this has been firmly decided yet. From the implementation here, I think we should embed the DV in manifest-specific metadata.
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
Review Comment:
These fields are required in v4, so they should be primitives in `ManifestStatsStruct`.
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
Review Comment:
These should be required in the DV struct, since the struct itself is optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539964591
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
Review Comment:
No need for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540033941
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
Review Comment:
It isn't safe to assign positions like this because this can't account for row group skipping. Instead, this should project the row position using the metadata column (adding `MetadataColumns.ROW_POSITION` in the schema) and updating the `set` method of `StructLike` to handle the position.
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
+ });
+ }
+
+ private static Function<TrackedFile<?>, TrackedFile<?>> rowIdAssigner(Long firstRowId) {
+ if (firstRowId == null) {
+ return entry -> entry;
+ }
+
+ return new Function<>() {
+ private long nextRowId = firstRowId;
+
+ @Override
+ public TrackedFile<?> apply(TrackedFile<?> entry) {
+ if (entry.contentType() == FileContent.DATA) {
+ TrackingInfo tracking = entry.trackingInfo();
+ if (tracking != null
+ && tracking.status() != TrackingInfo.Status.DELETED
+ && tracking.firstRowId() == null) {
+ entry.setFirstRowId(nextRowId);
+ nextRowId += entry.recordCount();
+ }
+ }
+ return entry;
+ }
+ };
+ }
+
+ private CloseableIterable<TrackedFile<?>> assignPositions(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.transform(
+ entries,
+ new Function<>() {
+ private long position = 0;
+
+ @Override
+ public TrackedFile<?> apply(TrackedFile<?> entry) {
+ entry.setPos(position++);
Review Comment:
We don't use the result of the `++` operator in Iceberg because it is confusing. It is much more clear to increment and then use the value or to use the value and then increment:
```java
entry.setPos(position);
position++;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539840457
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
+ Types.NestedField SNAPSHOT_ID =
+ Types.NestedField.optional(
+ 1,
+ "snapshot_id",
+ Types.LongType.get(),
+ "Snapshot ID where the file was added, or deleted if status is 2. Inherited when null.");
+ Types.NestedField SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 3,
+ "sequence_number",
+ Types.LongType.get(),
+ "Data sequence number of the file. Inherited when null and status is 1 (added). Must be equal to file_sequence_number if content_type is 3 or 4.");
+ Types.NestedField FILE_SEQUENCE_NUMBER =
+ Types.NestedField.optional(
+ 4,
+ "file_sequence_number",
+ Types.LongType.get(),
+ "File sequence number indicating when the file was added. Inherited when null and status is added. Must be equal to sequence_number if content_type is 3 or 4.");
+ Types.NestedField FIRST_ROW_ID =
+ Types.NestedField.optional(
+ 142,
+ "first_row_id",
+ Types.LongType.get(),
+ "The _row_id for the first row in the data file if content_type is 0. If content_type is 3, this is the starting _row_id to assign to rows added by ADDED data files.");
+
+ /**
+ * Returns the status of the entry.
+ *
+ * <p>Status values:
+ *
+ * <ul>
+ * <li>0: EXISTING - file was already in the table
+ * <li>1: ADDED - file newly added
+ * <li>2: DELETED - file removed
+ * </ul>
+ */
+ Status status();
+
+ /**
+ * Returns the snapshot ID where the file was added or deleted.
+ *
+ * <p>Inherited when null.
+ */
+ Long snapshotId();
+
+ /**
+ * Returns the data sequence number of the file.
+ *
+ * <p>Inherited when null and status is 1 (added). Must be equal to file_sequence_number if
+ * content_type is 3 or 4.
+ */
+ Long sequenceNumber();
Review Comment:
Since this is a new interface, should we use `dataSequenceNumber` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539838289
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
+ EXISTING(0),
+ ADDED(1),
+ DELETED(2);
+
+ private final int id;
+
+ Status(int id) {
+ this.id = id;
+ }
+
+ public int id() {
+ return id;
+ }
+ }
+
+ Types.NestedField STATUS =
+ Types.NestedField.required(0, "status", Types.IntegerType.get(), "Entry status");
+ Types.NestedField SNAPSHOT_ID =
+ Types.NestedField.optional(
+ 1,
+ "snapshot_id",
+ Types.LongType.get(),
+ "Snapshot ID where the file was added, or deleted if status is 2. Inherited when null.");
Review Comment:
I think this should be `"Snapshot ID where the file was added or deleted/removed"`. No need to mention specific status codes or inheritance. People reading these docs should never see values without a snapshot ID because of inheritance so I don't think there is value in adding it here because this is primarily for metadata tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539922319
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
+ }
+
+ @Override
+ public Long pos() {
+ return trackedFile.pos();
+ }
+
+ @Override
+ public int specId() {
+ return trackedFile.partitionSpecId();
+ }
+
+ @Override
+ public FileContent content() {
+ return trackedFile.contentType();
+ }
+
+ @Override
+ public CharSequence path() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public String location() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public FileFormat format() {
+ return trackedFile.fileFormat();
+ }
+
+ @Override
+ public StructLike partition() {
+ return null;
Review Comment:
This should be marked with a TODO. Implementing this on top of content stats is a key part of validating the v4 partition design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539910058
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadataFactory.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+
+/** Factory for creating {@link InheritableTrackedMetadata} instances. */
+class InheritableTrackedMetadataFactory {
+
+ private InheritableTrackedMetadataFactory() {}
+
+ /**
+ * Creates inheritable metadata from explicit values.
+ *
+ * @param snapshotId the snapshot ID
+ * @param sequenceNumber the sequence number
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata create(long snapshotId, long sequenceNumber) {
+ return new BaseInheritableTrackedMetadata(snapshotId, sequenceNumber);
+ }
+
+ /**
+ * Creates inheritable metadata from a tracked file representing a manifest (for reading leaf
+ * manifests).
+ *
+ * @param manifestEntry the DATA_MANIFEST or DELETE_MANIFEST tracked file from root
+ * @return inheritable metadata instance
+ */
+ static InheritableTrackedMetadata fromTrackedFile(TrackedFile<?> manifestEntry) {
+ Preconditions.checkArgument(
+ manifestEntry.contentType() == FileContent.DATA_MANIFEST
+ || manifestEntry.contentType() == FileContent.DELETE_MANIFEST,
+ "Can only create metadata from manifest entries, got: %s",
+ manifestEntry.contentType());
+
+ TrackingInfo tracking = manifestEntry.trackingInfo();
+ Long snapshotId = tracking != null ? tracking.snapshotId() : null;
+ Long sequenceNumber = tracking != null ? tracking.sequenceNumber() : null;
+
+ Preconditions.checkArgument(
+ snapshotId != null, "Manifest entry must have snapshot ID: %s", manifestEntry.location());
+
+ return new BaseInheritableTrackedMetadata(
+ snapshotId, sequenceNumber != null ? sequenceNumber : 0L);
+ }
+
+ static class BaseInheritableTrackedMetadata implements InheritableTrackedMetadata {
+ private final long snapshotId;
+ private final long sequenceNumber;
+
+ private BaseInheritableTrackedMetadata(long snapshotId, long sequenceNumber) {
+ this.snapshotId = snapshotId;
+ this.sequenceNumber = sequenceNumber;
Review Comment:
I think that this should also track the location of the manifest and update `TrackingInfo` with it. This is used at commit time to know where a given data file came from so that we can rewrite the containing manifest (v3) or update a MDV (v4) without searching for the containing manifest (unless it has been rewritten or removed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540025293
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
Review Comment:
In v4, we are inverting the relationship between an entry and a file. Now, a tracked file contains its "entry" metadata as tracking info. The purpose of that change is to avoid needing additional methods to get the "entries" versus getting the "files".
The reader can now be simplified. It needs two read capabilities: `allFiles` and `liveFiles` (which filters out removed/deleted files). The `ClosableIterable` interface should use `liveFiles` because that's the simplest generic contract: when a manifest is read you don't return deleted files unless they are requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683251726
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
Review Comment:
Moved them to `ManifestStatsStruct`, and made them primitives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2684528899
##########
core/src/main/java/org/apache/iceberg/TrackedDataFileAdapter.java:
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Adapter that wraps a TrackedFile and presents it as a DataFile.
+ *
+ * <p>This adapter allows TrackedFile instances to be used in contexts that expect DataFile. The
+ * adapter returns null for partition data and column-level statistics until ContentStats is
+ * implemented.
+ *
+ * <p>TODO: When ContentStats is implemented, use spec to extract partition from contentStats.
+ */
+class TrackedDataFileAdapter implements DataFile {
+ private final TrackedFile<?> trackedFile;
+
+ @SuppressWarnings("UnusedVariable")
+ private final PartitionSpec spec;
+
+ TrackedDataFileAdapter(TrackedFile<?> trackedFile, PartitionSpec spec) {
+ if (trackedFile.contentType() != FileContent.DATA) {
+ throw new IllegalStateException(
+ "Cannot convert TrackedFile with content type "
+ + trackedFile.contentType()
+ + " to DataFile");
+ }
+ this.trackedFile = trackedFile;
+ this.spec = spec;
+ }
+
+ @Override
+ public String manifestLocation() {
+ return trackedFile.manifestLocation();
+ }
+
+ @Override
+ public Long pos() {
+ return trackedFile.pos();
+ }
+
+ @Override
+ public int specId() {
+ return trackedFile.partitionSpecId();
+ }
+
+ @Override
+ public FileContent content() {
+ return trackedFile.contentType();
+ }
+
+ @Override
+ public CharSequence path() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public String location() {
+ return trackedFile.location();
+ }
+
+ @Override
+ public FileFormat format() {
+ return trackedFile.fileFormat();
+ }
+
+ @Override
+ public StructLike partition() {
+ return null;
+ }
+
+ @Override
+ public long recordCount() {
+ return trackedFile.recordCount();
+ }
+
+ @Override
+ public long fileSizeInBytes() {
+ Long size = trackedFile.fileSizeInBytes();
+ return size != null ? size : 0L;
+ }
+
+ @Override
+ public Map<Integer, Long> columnSizes() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> valueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> nullValueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, Long> nanValueCounts() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, ByteBuffer> lowerBounds() {
+ return null;
+ }
+
+ @Override
+ public Map<Integer, ByteBuffer> upperBounds() {
+ return null;
+ }
+
+ @Override
+ public ByteBuffer keyMetadata() {
+ return trackedFile.keyMetadata();
+ }
+
+ @Override
+ public List<Long> splitOffsets() {
+ return trackedFile.splitOffsets();
+ }
+
+ @Override
+ public List<Integer> equalityFieldIds() {
+ return null;
+ }
+
+ @Override
+ public Integer sortOrderId() {
+ return trackedFile.sortOrderId();
+ }
+
+ @Override
+ public Long dataSequenceNumber() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.sequenceNumber() : null;
+ }
+
+ @Override
+ public Long fileSequenceNumber() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.fileSequenceNumber() : null;
+ }
+
+ @Override
+ public Long firstRowId() {
+ TrackingInfo trackingInfo = trackedFile.trackingInfo();
+ return trackingInfo != null ? trackingInfo.firstRowId() : null;
+ }
+
+ @Override
+ public DataFile copy() {
+ return new TrackedDataFileAdapter((TrackedFile<?>) trackedFile.copy(), spec);
+ }
+
+ @Override
+ public DataFile copyWithoutStats() {
+ return new TrackedDataFileAdapter((TrackedFile<?>) trackedFile.copyWithoutStats(), spec);
+ }
+
+ @Override
+ public DataFile copyWithStats(Set<Integer> requestedColumnIds) {
+ return new TrackedDataFileAdapter(
+ (TrackedFile<?>) trackedFile.copyWithStats(requestedColumnIds), spec);
Review Comment:
It was unrelated to the type param. Fixed it with a refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683375598
##########
core/src/main/java/org/apache/iceberg/InheritableTrackedMetadata.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+
+/**
+ * Interface for applying inheritable metadata to tracked file entries in V4 manifests.
+ *
+ * <p>Similar to {@link InheritableMetadata} but for V4 TrackedFile entries.
+ */
+interface InheritableTrackedMetadata extends Serializable {
+ /**
+ * Apply inheritable metadata to a tracked file entry.
+ *
+ * @param entry the tracked file entry
+ * @return the entry with metadata applied
+ */
+ TrackedFile<?> apply(TrackedFile<?> entry);
Review Comment:
Thanks. I've implemented something along the lines of your suggestion. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-4264379662
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2678073344
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
Review Comment:
Thanks. That simplifies things a fair bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2683275833
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
+ Serializable {
+
+ private static final FileContent[] FILE_CONTENT_VALUES = FileContent.values();
+
+ // Tracking metadata
+ private String manifestLocation = null;
+ private TrackingInfo.Status status = null;
+ private Long snapshotId = null;
+ private Long sequenceNumber = null;
+ private Long fileSequenceNumber = null;
+ private Long firstRowId = null;
+ private Long position = null;
+
+ // Common file metadata
+ private FileContent contentType = FileContent.DATA;
+ private String location = null;
+ private FileFormat fileFormat = null;
+ private long recordCount = -1L;
+ private Long fileSizeInBytes = null;
+
+ // Deletion vector fields
+ private Long deletionVectorOffset = null;
+ private Long deletionVectorSizeInBytes = null;
+ private byte[] deletionVectorInlineContent = null;
+
+ // Partition and ordering
+ private int partitionSpecId = -1;
+ private Integer sortOrderId = null;
+
+ // Optional metadata
+ private byte[] keyMetadata = null;
+ private long[] splitOffsets = null;
+ private int[] equalityIds = null;
+ private String referencedFile = null;
+
+ // Manifest stats (for manifest entries)
+ private Integer addedFilesCount = null;
+ private Integer existingFilesCount = null;
+ private Integer deletedFilesCount = null;
+ private Long addedRowsCount = null;
+ private Long existingRowsCount = null;
+ private Long deletedRowsCount = null;
+ private Long minSequenceNumber = null;
+
+ // Content stats placeholder (TODO: implement ContentStats)
+ private Object contentStats = null;
+
+ // Cached schema for Avro
+ private transient Schema avroSchema = null;
+
+ // Base type that corresponds to positions for get/set
+ // All fields are made optional to support different content types
+ static final Types.StructType BASE_TYPE =
+ Types.StructType.of(
+ TrackedFile.CONTENT_TYPE,
+ TrackedFile.LOCATION,
+ TrackedFile.FILE_FORMAT,
+ TrackedFile.PARTITION_SPEC_ID,
+ TrackedFile.SORT_ORDER_ID,
+ TrackedFile.RECORD_COUNT,
+ TrackedFile.FILE_SIZE_IN_BYTES,
+ TrackedFile.KEY_METADATA,
+ TrackedFile.SPLIT_OFFSETS,
+ TrackedFile.EQUALITY_IDS,
+ TrackedFile.REFERENCED_FILE,
+ // Tracking info fields
+ TrackingInfo.STATUS.asOptional(),
+ TrackingInfo.SNAPSHOT_ID,
+ TrackingInfo.SEQUENCE_NUMBER,
+ TrackingInfo.FILE_SEQUENCE_NUMBER,
+ TrackingInfo.FIRST_ROW_ID,
+ // Deletion vector fields
+ DeletionVector.OFFSET,
+ DeletionVector.SIZE_IN_BYTES,
+ DeletionVector.INLINE_CONTENT,
+ // Manifest stats fields
+ ManifestStats.ADDED_FILES_COUNT.asOptional(),
+ ManifestStats.EXISTING_FILES_COUNT.asOptional(),
+ ManifestStats.DELETED_FILES_COUNT.asOptional(),
+ ManifestStats.ADDED_ROWS_COUNT.asOptional(),
+ ManifestStats.EXISTING_ROWS_COUNT.asOptional(),
+ ManifestStats.DELETED_ROWS_COUNT.asOptional(),
+ ManifestStats.MIN_SEQUENCE_NUMBER.asOptional());
+
+ /** Used by Avro reflection to instantiate this class when reading manifest files. */
+ public GenericTrackedFile(Schema avroSchema) {
+ this(AvroSchemaUtil.convert(avroSchema).asStructType());
+ this.avroSchema = avroSchema;
+ }
+
+ /** Used by internal readers to instantiate this class with a projection schema. */
+ GenericTrackedFile(Types.StructType projection) {
+ super(BASE_TYPE, projection);
+ this.avroSchema = AvroSchemaUtil.convert(projection, "tracked_file");
+ }
+
+ public GenericTrackedFile() {
+ super(BASE_TYPE.fields().size());
+ }
+
+ /**
+ * Full constructor for creating a tracked file.
+ *
+ * @param contentType the type of content (DATA, POSITION_DELETES, etc.)
+ * @param location the file location
+ * @param fileFormat the file format
+ * @param partitionSpecId the partition spec ID
+ * @param recordCount the number of records
+ */
+ public GenericTrackedFile(
Review Comment:
Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520523382
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
Review Comment:
I think that this also requires a field to be defined, so we have a record of the ID used for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520516492
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
Review Comment:
Same here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520512362
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
Review Comment:
Not using a separate entry for inline would make this required, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2529459895
##########
api/src/main/java/org/apache/iceberg/DeletionVector.java:
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Deletion vector information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This encapsulates both inline and out-of-line deletion vector content. This struct must be
+ * defined for types 1 (POSITION_DELETES) and 5 (MANIFEST_DV).
+ */
+public interface DeletionVector {
+ Types.NestedField OFFSET =
+ Types.NestedField.optional(
+ 144, "offset", Types.LongType.get(), "Offset in the file where the content starts");
+ Types.NestedField SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 145,
+ "size_in_bytes",
+ Types.LongType.get(),
+ "Length of a referenced content stored in the file; required if offset is present");
+ Types.NestedField INLINE_CONTENT =
+ Types.NestedField.optional(
+ 146, "inline_content", Types.BinaryType.get(), "Serialized bitmap for inline DVs");
+
+ /**
+ * Returns the offset in the file where the deletion vector content starts.
+ *
+ * <p>This is used for out-of-line deletion vectors stored in Puffin files.
+ */
+ Long offset();
+
+ /**
+ * Returns the size in bytes of the deletion vector content.
+ *
+ * <p>Required if offset is present.
+ */
+ Long sizeInBytes();
+
+ /**
+ * Returns the serialized bitmap for inline deletion vectors.
+ *
+ * <p>When present, the deletion vector is stored inline in the manifest rather than in a separate
+ * Puffin file.
+ */
+ ByteBuffer inlineContent();
Review Comment:
Data and metadata DVs seemed like similar concepts to me. But I don't mind changing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539958965
##########
core/src/main/java/org/apache/iceberg/GenericTrackedFile.java:
##########
@@ -0,0 +1,773 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.Serializable;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Locale;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.avro.specific.SpecificData;
+import org.apache.iceberg.avro.AvroSchemaUtil;
+import org.apache.iceberg.avro.SupportsIndexProjection;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.ArrayUtil;
+import org.apache.iceberg.util.ByteBuffers;
+
+/** Generic implementation of {@link TrackedFile} for V4 manifests. */
+public class GenericTrackedFile extends SupportsIndexProjection
+ implements TrackedFile<GenericTrackedFile>,
+ IndexedRecord,
+ StructLike,
+ SpecificData.SchemaConstructable,
Review Comment:
There's no need for Avro interfaces since we no longer use Avro generics to read metadata. You can remove `IndexedRecord` because it is replaced by `StructLike` and remove `SpecificData.SchemaConstructable` because that's handled by [reflection in the readers](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/InternalReaders.java#L56-L59).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2539889521
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
+ Types.NestedField KEY_METADATA =
+ Types.NestedField.optional(
+ 131,
+ "key_metadata",
+ Types.BinaryType.get(),
+ "Implementation-specific key metadata for encryption");
+ Types.NestedField SPLIT_OFFSETS =
+ Types.NestedField.optional(
+ 132,
+ "split_offsets",
+ Types.ListType.ofRequired(133, Types.LongType.get()),
+ "Split offsets for the data file. Must be sorted ascending");
+ Types.NestedField EQUALITY_IDS =
+ Types.NestedField.optional(
+ 135,
+ "equality_ids",
+ Types.ListType.ofRequired(136, Types.IntegerType.get()),
+ "Field ids used to determine row equality in equality delete files. Required when content=2");
+ Types.NestedField REFERENCED_FILE =
+ Types.NestedField.optional(
+ 143,
+ "referenced_file",
+ Types.StringType.get(),
+ "Location of data file that a DV references if content_type is 1 or 5."
+ + " Location of affiliated data manifest if content_type is 4 or null if delete manifest is unaffiliated");
+
+ /**
+ * Returns the path of the manifest which this file is referenced in or null if it was not read
+ * from a manifest.
+ */
+ String manifestLocation();
+
+ /**
+ * Returns the tracking information for this entry.
+ *
+ * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+ * tracking info is inherited.
+ */
+ TrackingInfo trackingInfo();
+
+ /**
+ * Returns the type of content stored by this entry.
+ *
+ * <p>One of: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
+ FileContent contentType();
+
+ /**
+ * Returns the location of the file.
+ *
+ * <p>Optional if content_type is MANIFEST_DV and deletion_vector has inline content.
+ */
+ String location();
+
+ /** Returns the format of the file (avro, orc, parquet, or puffin). */
+ FileFormat fileFormat();
+
+ /**
+ * Returns the deletion vector for this entry.
+ *
+ * <p>Must be defined if content_type is POSITION_DELETES or MANIFEST_DV. Must be null for all
+ * other types.
+ */
+ DeletionVector deletionVector();
+
+ /** Returns the ID of the partition spec used to write this file or manifest. */
+ int partitionSpecId();
+
+ /**
+ * Returns the ID representing sort order for this file.
+ *
+ * <p>Can only be set if content_type is DATA.
+ */
+ Integer sortOrderId();
+
+ /** Returns the number of records in this file, or the cardinality of a deletion vector. */
+ long recordCount();
+
+ /**
+ * Returns the total file size in bytes.
+ *
+ * <p>Must be defined if location is defined.
+ */
+ Long fileSizeInBytes();
+
+ /**
+ * Returns the content stats for this entry.
+ *
+ * <p>TODO: Define ContentStats structure per V4 proposal.
+ */
+ Object contentStats();
+
+ /**
+ * Returns the manifest stats for this entry.
+ *
+ * <p>Must be set if content_type is DATA_MANIFEST or DELETE_MANIFEST, otherwise must be null.
+ */
+ ManifestStats manifestStats();
+
+ /** Returns metadata about how this file is encrypted, or null if stored in plain text. */
+ ByteBuffer keyMetadata();
+
+ /**
+ * Returns list of recommended split locations, if applicable, null otherwise.
+ *
+ * <p>Must be sorted in ascending order.
+ */
+ List<Long> splitOffsets();
+
+ /**
+ * Returns the set of field IDs used for equality comparison, in equality delete files.
+ *
+ * <p>Required when content_type is EQUALITY_DELETES, must be null otherwise.
+ */
+ List<Integer> equalityIds();
+
+ /**
+ * Returns the location of the referenced file.
+ *
+ * <p>For POSITION_DELETES or MANIFEST_DV: location of data file that the DV references.
+ *
+ * <p>For DELETE_MANIFEST: location of affiliated data manifest, or null if unaffiliated.
+ */
+ String referencedFile();
+
+ /**
+ * Copies this tracked file.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data when collecting
+ * files from tasks.
+ */
+ F copy();
+
+ /**
+ * Copies this tracked file without stats.
+ *
+ * <p>Use this method to copy data without stats when collecting files.
+ */
+ F copyWithoutStats();
+
+ /**
+ * Copies this tracked file with stats only for specific columns.
+ *
+ * <p>Manifest readers can reuse file instances; use this method to copy data with stats only for
+ * specific columns when collecting files.
+ *
+ * @param requestedColumnIds column IDs for which to keep stats
+ * @return a copy of this tracked file, with content stats for only the requested columns
+ */
+ F copyWithStats(Set<Integer> requestedColumnIds);
+
+ /**
+ * Returns the ordinal position in the manifest.
+ *
+ * <p>Used for applying manifest deletion vectors.
+ */
+ Long pos();
+
+ /**
+ * Converts this tracked file to a DataFile.
+ *
+ * <p>Only valid when content_type is DATA. Extracts partition from contentStats using the
+ * provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DataFile representation
+ * @throws IllegalStateException if content_type is not DATA
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DataFile asDataFile(PartitionSpec spec);
+
+ /**
+ * Converts this tracked file to a DeleteFile.
+ *
+ * <p>Only valid when content_type is POSITION_DELETES or EQUALITY_DELETES. Extracts partition
+ * from contentStats using the provided spec.
+ *
+ * @param spec the partition spec (needed to know which transforms to apply)
+ * @return a DeleteFile representation
+ * @throws IllegalStateException if content_type is not a delete type
+ * @throws UnsupportedOperationException if ContentStats not yet implemented
+ */
+ DeleteFile asDeleteFile(PartitionSpec spec);
+
+ /** Set the status for this tracked file entry. */
+ void setStatus(TrackingInfo.Status status);
+
+ /** Set the snapshot ID for this tracked file entry. */
+ void setSnapshotId(Long snapshotId);
+
+ /** Set the data sequence number for this tracked file entry. */
+ void setSequenceNumber(Long sequenceNumber);
+
+ /** Set the file sequence number for this tracked file entry. */
+ void setFileSequenceNumber(Long fileSequenceNumber);
+
+ /** Set the first row ID for this tracked file entry. */
+ void setFirstRowId(Long firstRowId);
+
+ /** Set the ordinal position in the manifest. */
+ void setPos(Long position);
Review Comment:
I don't think that this is needed. In the v3 implementation, we use a column that projects _pos without calling a setter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540029564
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
Review Comment:
All filtering should happen before modifying rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2540069300
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.Function;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.io.CloseableGroup;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.CloseableIterator;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.roaringbitmap.RoaringBitmap;
+
+/**
+ * Reader for V4 manifest files containing TrackedFile entries.
+ *
+ * <p>Supports reading both root manifests and leaf manifests. Returns TrackedFile entries which can
+ * represent data files, delete files, or manifest references. TODO: implement caching.
+ */
+class V4ManifestReader extends CloseableGroup implements CloseableIterable<TrackedFile<?>> {
+ static final ImmutableList<String> ALL_COLUMNS = ImmutableList.of("*");
+
+ private final InputFile file;
+ private final InheritableTrackedMetadata inheritableMetadata;
+ private final Long manifestFirstRowId;
+
+ private Collection<String> columns = null;
+ private TrackedFile<?> manifestDV = null;
+
+ protected V4ManifestReader(
+ InputFile file, InheritableTrackedMetadata inheritableMetadata, Long manifestFirstRowId) {
+ this.file = file;
+ this.inheritableMetadata = inheritableMetadata;
+ this.manifestFirstRowId = manifestFirstRowId;
+ }
+
+ public V4ManifestReader select(Collection<String> newColumns) {
+ this.columns = newColumns;
+ return this;
+ }
+
+ public V4ManifestReader withDeletionVector(TrackedFile<?> dv) {
+ this.manifestDV = dv;
+ return this;
+ }
+
+ public CloseableIterable<TrackedFile<?>> entries() {
+ return entries(false);
+ }
+
+ public CloseableIterable<TrackedFile<?>> liveEntries() {
+ return entries(true);
+ }
+
+ private CloseableIterable<TrackedFile<?>> entries(boolean onlyLive) {
+ CloseableIterable<TrackedFile<?>> entries = open(columns);
+ return onlyLive ? filterLiveEntries(entries) : entries;
+ }
+
+ private CloseableIterable<TrackedFile<?>> open(Collection<String> cols) {
+ Schema projection = buildProjection(cols);
+
+ FileFormat format = FileFormat.fromFileName(file.location());
+
+ CloseableIterable<GenericTrackedFile> entries =
+ InternalData.read(format, file)
+ .project(projection)
+ .setRootType(GenericTrackedFile.class)
+ .build();
+
+ addCloseable(entries);
+
+ CloseableIterable<TrackedFile<?>> transformed =
+ CloseableIterable.transform(entries, inheritableMetadata::apply);
+
+ transformed = CloseableIterable.transform(transformed, rowIdAssigner(manifestFirstRowId));
+
+ transformed = assignPositions(transformed);
+
+ if (manifestDV != null) {
+ RoaringBitmap deletedPositions = deserializeManifestDV(manifestDV);
+ transformed =
+ CloseableIterable.filter(
+ transformed,
+ entry -> {
+ Long pos = entry.pos();
+ // positions are 0-based and should not exceed Integer.MAX_VALUE
+ return pos == null || !deletedPositions.contains(pos.intValue());
+ });
+ }
+
+ return transformed;
+ }
+
+ private static RoaringBitmap deserializeManifestDV(TrackedFile<?> manifestDV) {
+ Preconditions.checkArgument(
+ manifestDV.contentType() == FileContent.MANIFEST_DV,
+ "Expected MANIFEST_DV, got: %s",
+ manifestDV.contentType());
+
+ DeletionVector dvInfo = manifestDV.deletionVector();
+ Preconditions.checkNotNull(dvInfo, "MANIFEST_DV must have deletion_vector");
+
+ Preconditions.checkNotNull(
+ dvInfo.inlineContent(),
+ "Manifest DV must have inline content (External not supported): %s",
+ manifestDV.referencedFile());
+
+ ByteBuffer buffer = dvInfo.inlineContent();
+ byte[] bytes = new byte[buffer.remaining()];
+ buffer.asReadOnlyBuffer().get(bytes);
+
+ RoaringBitmap bitmap = new RoaringBitmap();
+ try {
+ bitmap.deserialize(new DataInputStream(new ByteArrayInputStream(bytes)));
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to deserialize Roaring bitmap from manifest DV");
+ }
+ return bitmap;
+ }
+
+ private Schema buildProjection(Collection<String> cols) {
+ if (cols == null || cols.containsAll(ALL_COLUMNS)) {
+ return new Schema(GenericTrackedFile.BASE_TYPE.fields());
+ }
+
+ List<Types.NestedField> fields = Lists.newArrayList();
+ for (String column : cols) {
+ Types.NestedField field = GenericTrackedFile.BASE_TYPE.field(column);
+ if (field != null) {
+ fields.add(field);
+ }
+ }
+
+ return new Schema(fields);
+ }
+
+ private CloseableIterable<TrackedFile<?>> filterLiveEntries(
+ CloseableIterable<TrackedFile<?>> entries) {
+ return CloseableIterable.filter(
+ entries,
+ entry -> {
+ TrackingInfo tracking = entry.trackingInfo();
+ return tracking == null || tracking.status() != TrackingInfo.Status.DELETED;
Review Comment:
I think that tracking should never be null here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2529458863
##########
api/src/main/java/org/apache/iceberg/FileContent.java:
##########
@@ -18,11 +18,27 @@
*/
package org.apache.iceberg;
-/** Content type stored in a file, one of DATA, POSITION_DELETES, or EQUALITY_DELETES. */
+/**
+ * Content type stored in a file.
+ *
+ * <p>For V1-V3 tables: DATA, POSITION_DELETES, or EQUALITY_DELETES.
+ *
+ * <p>For V4 tables: DATA, POSITION_DELETES, EQUALITY_DELETES, DATA_MANIFEST, DELETE_MANIFEST, or
+ * MANIFEST_DV.
+ */
public enum FileContent {
DATA(0),
POSITION_DELETES(1),
- EQUALITY_DELETES(2);
+ EQUALITY_DELETES(2),
+ /** Data manifest entry (V4+ only) - references data files in a root manifest. */
+ DATA_MANIFEST(3),
+ /** Delete manifest entry (V4+ only) - references delete files in a root manifest. */
+ DELETE_MANIFEST(4),
+ /**
+ * Manifest deletion vector entry (V4+ only) - marks entries in a manifest as deleted without
+ * rewriting the manifest.
+ */
+ MANIFEST_DV(5);
Review Comment:
That was my preference too, and I advocated for it in our community discussion but this is what we settled on. Our current v4 proposal specifically uses `MANIFEST_DV` as a separate content type that references manifests via the referenced_file field. We can certainly change it and want to get take from others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2520537712
##########
api/src/main/java/org/apache/iceberg/TrackingInfo.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import org.apache.iceberg.types.Types;
+
+/**
+ * Tracking information for a tracked file entry in a V4 manifest.
+ *
+ * <p>This groups the status, snapshot, and sequence number information for the entry. This enables
+ * accessing the fields for the entry and provides an isolated structure that can be modified.
+ */
+public interface TrackingInfo {
+ /** Status of an entry in a tracked file */
+ enum Status {
Review Comment:
Isn't this enum already defined somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#discussion_r2674094987
##########
api/src/main/java/org/apache/iceberg/TrackedFile.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.nio.ByteBuffer;
+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/**
+ * Represents a V4 content entry in a manifest file.
+ *
+ * <p>TrackedFile is the V4 equivalent of ContentFile. It provides a unified representation for all
+ * entry types in a V4 manifest: data files, delete files, manifests, and deletion vectors.
+ *
+ * @param <F> the concrete class of a TrackedFile instance
+ */
+public interface TrackedFile<F> {
+ // Field IDs from V4 specification
+ Types.NestedField CONTENT_TYPE =
+ Types.NestedField.required(
+ 134,
+ "content_type",
+ Types.IntegerType.get(),
+ "Type of content: 0=DATA, 1=POSITION_DELETES, 2=EQUALITY_DELETES, 3=DATA_MANIFEST, 4=DELETE_MANIFEST, 5=MANIFEST_DV");
+ Types.NestedField LOCATION =
+ Types.NestedField.optional(
+ 100,
+ "location",
+ Types.StringType.get(),
+ "Location of the file. Optional if content_type is 5 and deletion_vector.inline_content is not null");
+ Types.NestedField FILE_FORMAT =
+ Types.NestedField.required(
+ 101,
+ "file_format",
+ Types.StringType.get(),
+ "String file format name: avro, orc, parquet, or puffin");
+ Types.NestedField PARTITION_SPEC_ID =
+ Types.NestedField.required(
+ 148,
+ "partition_spec_id",
+ Types.IntegerType.get(),
+ "ID of partition spec used to write manifest or data/delete files");
+ Types.NestedField SORT_ORDER_ID =
+ Types.NestedField.optional(
+ 140,
+ "sort_order_id",
+ Types.IntegerType.get(),
+ "ID representing sort order for this file. Can only be set if content_type is 0");
+ Types.NestedField RECORD_COUNT =
+ Types.NestedField.required(
+ 103,
+ "record_count",
+ Types.LongType.get(),
+ "Number of records in this file, or the cardinality of a deletion vector");
+ Types.NestedField FILE_SIZE_IN_BYTES =
+ Types.NestedField.optional(
+ 104,
+ "file_size_in_bytes",
+ Types.LongType.get(),
+ "Total file size in bytes. Must be defined if location is defined");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [WIP] V4 Manifest Read Support [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14533:
URL: https://github.com/apache/iceberg/pull/14533#issuecomment-4043054116
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org