You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stevenzwu (via GitHub)" <gi...@apache.org> on 2026/05/11 21:20:52 UTC
Re: [PR] Spec: V4 Adaptive Metadata Tree Spec Changes for Entry Structures [iceberg]
stevenzwu commented on code in PR #16025:
URL: https://github.com/apache/iceberg/pull/16025#discussion_r3197194239
##########
format/spec.md:
##########
@@ -130,8 +130,10 @@ Tables do not require rename, except for tables that use atomic rename to implem
* **Schema** -- Names and types of fields in a table.
* **Partition spec** -- A definition of how partition values are derived from data fields.
* **Snapshot** -- The state of a table at some point in time, including the set of all data files.
-* **Manifest list** -- A file that lists manifest files; one per snapshot.
-* **Manifest** -- A file that lists data or delete files; a subset of a snapshot.
+* **Manifest list** -- (V1-V3 only) A file that lists manifest files; one per snapshot.
Review Comment:
nit: most of the time, lower case `v1`, `v2` are used in the spec
##########
format/spec.md:
##########
@@ -130,8 +130,10 @@ Tables do not require rename, except for tables that use atomic rename to implem
* **Schema** -- Names and types of fields in a table.
* **Partition spec** -- A definition of how partition values are derived from data fields.
* **Snapshot** -- The state of a table at some point in time, including the set of all data files.
-* **Manifest list** -- A file that lists manifest files; one per snapshot.
-* **Manifest** -- A file that lists data or delete files; a subset of a snapshot.
+* **Manifest list** -- (V1-V3 only) A file that lists manifest files; one per snapshot.
+* **Root Manifest** -- (V4+) A manifest that can reference data files, delete files, and other data and delete manifests; one per snapshot. Replaces manifest lists in V4.
+* **Data manifest** -- A file that lists data files; a subset of a snapshot.
Review Comment:
Do we want to want to mention colocated DVs and column files for data manifests in v4+?
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
Review Comment:
this sentence reads like only v1-v3 stores these metadata as properties. but later, we also listed properties for v4. so the wording probably needs some clarification.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
Review Comment:
POSITION_DELETES handling — the table omits value 1 entirely, but the comment two paragraphs below ("Value 1 (POSITION_DELETES) no longer applies in entries") is the only place that says so. Worth making the table itself unambiguous: either list `1: RESERVED` (writers must not produce; readers must reject) or call out in the table cell that 1 is intentionally skipped. Otherwise a reader reaches for the int and wonders if the spec just forgot it.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
Review Comment:
the `deletion_vector` schema is only for external data DVs with `location` field. the **inline** manifest DV is stored as part of the `manifest_info`, which is a bit inconsistent to me. Ideally, the DV struct can capture both outline (data DV) and inline (manifest DV) at the top level. E.g. the `cardinality` field can be shared.
##########
format/spec.md:
##########
@@ -484,7 +486,7 @@ Note that:
### Partitioning
-Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. (Manifests store data files from any partition, as long as the partition spec is the same for the data files.)
+Data files are stored in manifests with partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. In V1-V3, manifests store data files from any partition, as long as the partition spec is the same for the data files. In V4, manifests can store data files from different partition specs because partition values are stored as column statistics.
Review Comment:
> because partition values are stored as column statistics.
This sentence probably needs to be updated based on the last community sync.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* | Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | *optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* | Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional* | Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* | Field ids for row equality in equality delete files. |
+
+ Value 1 (POSITION_DELETES) no longer applies in entries because deletion vector metadata is colocated with data files (`content_type` 0).
+
+ Leaf data manifests may only contain entries with `content_type` 0 (DATA); leaf delete manifests may only contain entries with `content_type` 2 (EQUALITY DELETES).
Review Comment:
leaf delete manifest files can also contain entries with position deletes written back when the format version is 2. that also means the line 683 above is also inaccurate
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
Review Comment:
nit: wondering if `deletion_vector` and `column_files` should be added to the end of the list? other fields are about the base data files.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* | Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | *optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* | Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional* | Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* | Field ids for row equality in equality delete files. |
+
+ Value 1 (POSITION_DELETES) no longer applies in entries because deletion vector metadata is colocated with data files (`content_type` 0).
+
+ Leaf data manifests may only contain entries with `content_type` 0 (DATA); leaf delete manifests may only contain entries with `content_type` 2 (EQUALITY DELETES).
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is not 0.
+ - `deleted_positions` and `replaced_positions` must be null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3: REPLACED) | *required* | *required* | Used to track additions, deletions, and replacements. REPLACED indicates entries with data column updates or `deletion_vector` changes. Deleted entries are required when the snapshot has a non-null parent. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID where the file was added or deleted. Inherited when null. Optional for leaf manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID where the deletion vector was added. Inherited when null. Must be null when `deletion_vector` is null. |
+ | 3 | **`sequence_number`** | `long` | *optional* | *optional* | Data sequence number of the file. Inherited when null and status is 1 (ADDED). Must equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf manifests, required for root. |
+ | 4 | **`file_sequence_number`** | `long` | *optional* | *optional* | File sequence number indicating when the file was added. Inherited when null and status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. |
+ | 142 | **`first_row_id`** | `long` | *optional* | *optional* | The `_row_id` for the first row in the data file if `content_type` is 0. If `content_type` is 3, this is the starting `_row_id` to assign to rows added by ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | 6 | **`deleted_positions`** | `binary` | *optional* | *optional* | [TODO: link to roaring bitmap spec] Roaring bitmap of positions in the referenced leaf manifest that were deleted in this snapshot. |
Review Comment:
I don't quite understand these two fields `deleted_positions` and `replaced_positions`.
* is `deleted_positions` the diff manifest DV that we talked about?
* Is `replaced_positions` the old manifest DV? if yes, why do we need it? We just need the new manifest DV and the diff manifest DV.
I have been also thinking about change detection for the data DVs. We don't store the references to the old data DV in the new snapshot. that means snapshot diff for change detection.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* | Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | *optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* | Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional* | Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* | Field ids for row equality in equality delete files. |
+
+ Value 1 (POSITION_DELETES) no longer applies in entries because deletion vector metadata is colocated with data files (`content_type` 0).
+
+ Leaf data manifests may only contain entries with `content_type` 0 (DATA); leaf delete manifests may only contain entries with `content_type` 2 (EQUALITY DELETES).
+
+ The following constraints apply based on `content_type`:
Review Comment:
it is better to capture these in the description column in the table (instead of a separate bullet list here).
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
Review Comment:
> The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
It might be fine just removing this sentence. it reads a bit weird especially `The V4 data file`. If we want to keep it, it is more clear to say `In V4, data_file struct fields are flattened ...`
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
Review Comment:
> V4 writers must produce `writer_format_version` 1.
why not 4 for V4 writers? null for entries written before V4.
Also, the `0: PRE-V4` enum is odd: PRE-V4 manifests use the `manifest_entry` schema, not `content_entry`, so they can't have this field. When does value 0 ever appear?
##########
format/spec.md:
##########
@@ -733,20 +832,20 @@ Any null (unassigned) `first_row_id` must be assigned via inheritance, even if t
### Snapshots
A snapshot consists of the following fields:
-
-| v1 | v2 | v3 | Field | Description |
-| ---------- | ---------- |------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | _required_ | **`snapshot-id`** | A unique long ID |
-| _optional_ | _optional_ | _optional_ | **`parent-snapshot-id`** | The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent |
-| | _required_ | _required_ | **`sequence-number`** | A monotonically increasing long that tracks the order of changes to a table |
-| _required_ | _required_ | _required_ | **`timestamp-ms`** | A timestamp when the snapshot was created, used for garbage collection and table inspection |
-| _optional_ | _required_ | _required_ | **`manifest-list`** | The location of a manifest list for this snapshot that tracks manifest files with additional metadata |
-| _optional_ | | | **`manifests`** | A list of manifest file locations. Must be omitted if `manifest-list` is present |
-| _optional_ | _required_ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) |
-| _optional_ | _optional_ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created |
-| | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) |
-| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
-| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+=== "v1 - v3"
Review Comment:
also need a table for `v4`
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* | Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | *optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* | Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional* | Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* | Field ids for row equality in equality delete files. |
+
+ Value 1 (POSITION_DELETES) no longer applies in entries because deletion vector metadata is colocated with data files (`content_type` 0).
+
+ Leaf data manifests may only contain entries with `content_type` 0 (DATA); leaf delete manifests may only contain entries with `content_type` 2 (EQUALITY DELETES).
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is not 0.
+ - `deleted_positions` and `replaced_positions` must be null when `content_type` is not 3 or 4.
Review Comment:
Cross-struct constraint reference: `deleted_positions` and `replaced_positions` are inside the `tracking` sub-struct (field IDs 6 and 7), not at the content_entry top level. Reading this constraint without already knowing that, it's unclear which fields it's talking about. Suggest either:
- Naming them with the struct prefix: "`tracking.deleted_positions` and `tracking.replaced_positions` must be null..."
- Or moving the constraint into the `tracking` struct's own description below.
Same applies to the `dv_snapshot_id`/`deletion_vector` cross-struct constraint at line 702 — making the struct boundary explicit will help V4 implementers.
##########
format/spec.md:
##########
@@ -590,85 +592,182 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A manifest is an immutable file that lists data files or delete files, along with each file’s partition data, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. In V1-V3, manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot. In V4, a single root manifest per snapshot can directly reference data files, delete files, and other data and delete manifests.
-A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
+Manifests are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection.
A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+**Partition Spec Binding:**
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
-
-| v1 | v2 | Key | Value |
-|------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
-| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
-| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
-| _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
-| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
-| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
-| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
-
-The schema of a manifest file is defined by the `manifest_entry` struct, described in the following section.
-
-#### Manifest Entry Fields
-
-The `manifest_entry` struct consists of the following fields:
-
-| v1 | v2 | Field id, name | Type | Description |
-| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
-| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
-| _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
-| | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
-| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
-| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
-
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+- V1-V3: A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec. The partition spec of each manifest is used to transform predicates on the table’s data rows into predicates on partition values during job planning.
+- V4: Manifests are not bound to a single partition spec. Files with different partition specs can coexist in the same manifest because partition values are stored in column statistics using source column IDs rather than in a partition-spec-specific struct. The `partition-spec-id` in manifest metadata is tracked for informational purposes but does not constrain the contents.
+
+#### Manifest File Format
+
+Manifests are Avro files in V1-V3. Starting in V4, writers must produce manifests in Parquet.
+
+In V1-V3, a manifest file must store the partition spec and other metadata as properties in the file’s key-value metadata:
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
+ | _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | _required_ | `partition-spec` | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](#partition-specs) |
+ | _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
+ | | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+=== "v4"
+ | Write | Read | Key | Value |
+ |------------|------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
+ | _optional_ | _optional_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _optional_ | _optional_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
+ | _optional_ | _optional_ | `format-version` | Table format version number of the manifest as a string |
+ | _optional_ | _optional_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+
+The schema of a manifest file is defined by the `manifest_entry` struct (V1-V3) or `content_entry` struct (V4), described in the following section.
+
+#### Entries in Manifests
+
+In V1-V3, manifest entries are described by the `manifest_entry` struct. In V4, entries are called content entries and are described by the `content_entry` struct. The V4 `data_file` struct fields are flattened directly into the content entry, and tracking fields are grouped into a nested `tracking` struct.
+
+=== "v1 - v3"
+ | v1 | v2 and v3 | Field id, name | Type | Description |
+ | ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
+ | _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
+ | _required_ | _optional_ | **`1 snapshot_id`** | `long` | Snapshot id where the file was added, or deleted if status is 2. Inherited when null. |
+ | | _optional_ | **`3 sequence_number`** | `long` | Data sequence number of the file. Inherited when null and status is 1 (added). |
+ | | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
+ | _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
+
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
+
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
+
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
+
+ Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
+ The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+ The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+ The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
+
+ Notes:
+
+ 1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
+ 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+
+=== "v4"
+ **Content Entries**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 134 | **`content_type`** | `int` (0: DATA, 2: EQUALITY DELETES, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. |
+ | 157 | **`writer_format_version`** | `int` (0: PRE-V4, 1: V4) | *required* | *required* | Writer format version. V4 writers must produce `writer_format_version` 1. |
+ | 100 | **`location`** | `string` | *required* | *required* | Location of the file or manifest. |
+ | 158 | **`column_files`** | `list<column_file>` | *optional* | *optional* | Column update files associated with this entry. |
+ | 101 | **`file_format`** | `string` | *required* | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` |
+ | 147 | **`tracking`** | `tracking` struct | *required* | *required* | Groups status, snapshot, and sequence number. See tracking struct below. |
+ | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | *optional* | Row-level deletion vector for a data file. |
+ | 141 | **`spec_id`** | `int` | *optional* | *optional* | ID of the partition spec used to write this manifest or data file. |
+ | 140 | **`sort_order_id`** | `int` | *optional* | *optional* | ID representing sort order for this file. |
+ | 103 | **`record_count`** | `long` | *required* | *required* | Number of records in this file. |
+ | 104 | **`file_size_in_bytes`** | `long` | *required* | *required* | Total file size in bytes. |
+ | 146 | **`content_stats`** | `content_stats` struct | *optional* | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). |
+ | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | *optional* | See manifest_info struct below. |
+ | 131 | **`key_metadata`** | `binary` | *optional* | *optional* | Implementation-specific key metadata for encryption. |
+ | 132 | **`split_offsets`** | `list<133: long>` | *optional* | *optional* | Split offsets for the data file. Must be sorted ascending. |
+ | 135 | **`equality_ids`** | `list<136: int>` | *optional* | *optional* | Field ids for row equality in equality delete files. |
+
+ Value 1 (POSITION_DELETES) no longer applies in entries because deletion vector metadata is colocated with data files (`content_type` 0).
+
+ Leaf data manifests may only contain entries with `content_type` 0 (DATA); leaf delete manifests may only contain entries with `content_type` 2 (EQUALITY DELETES).
+
+ The following constraints apply based on `content_type`:
+
+ - `manifest_info` must be set when `content_type` is 3 or 4; must be null otherwise.
+ - `deletion_vector` may only be set when `content_type` is 0; must be null otherwise.
+ - `equality_ids` must be set when `content_type` is 2; must be null otherwise.
+ - `column_files` must be null when `content_type` is not 0 or 3.
+ - `sort_order_id` must be null when `content_type` is not 0.
+ - `deleted_positions` and `replaced_positions` must be null when `content_type` is not 3 or 4.
+
+ **`tracking` struct (field 147)**
+
+ | Field id | Name | Type | Write | Read | Description |
+ |----------|------|------|-------|------|-------------|
+ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3: REPLACED) | *required* | *required* | Used to track additions, deletions, and replacements. REPLACED indicates entries with data column updates or `deletion_vector` changes. Deleted entries are required when the snapshot has a non-null parent. Deletes are not used in scans. |
+ | 1 | **`snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID where the file was added or deleted. Inherited when null. Optional for leaf manifests, required for root. |
+ | 5 | **`dv_snapshot_id`** | `long` | *optional* | *optional* | Snapshot ID where the deletion vector was added. Inherited when null. Must be null when `deletion_vector` is null. |
Review Comment:
is it better to move this one to the `deletion_vector` struct? the tracking is only for the content file (data or manifest).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org