You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by am...@apache.org on 2026/10/14 21:15:48 UTC
(iceberg) branch main updated: Spec: V4 Adaptive Metadata Tree Spec Changes for Entry Structures (#16025)
This is an automated email from the ASF dual-hosted git repository.
amogh-jahagirdar pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new fd3a81b15c Spec: V4 Adaptive Metadata Tree Spec Changes for Entry Structures (#16025)
fd3a81b15c is described below
commit fd3a81b15c1b3d0741ac3937f9c2c29567bd0406
Author: Amogh Jahagirdar <am...@apache.org>
AuthorDate: Wed Oct 14 15:15:42 2026 -0600
Spec: V4 Adaptive Metadata Tree Spec Changes for Entry Structures (#16025)
Co-authored-by: Gabor Kaszab <ga...@gmail.com>
Co-authored-by: Anurag Mantripragada <am...@apple.com>
Co-authored-by: Ryan Blue <bl...@apache.org>
Co-authored-by: Russell Spitzer <ru...@GMAIL.COM>
Co-authored-by: Steven Zhen Wu <st...@gmail.com>
Co-authored-by: pvary <pe...@gmail.com>
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
format/spec.md | 333 ++++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 256 insertions(+), 77 deletions(-)
diff --git a/format/spec.md b/format/spec.md
index 844c2b2fb6..3e8ba7232f 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -61,6 +61,8 @@ The full set of changes are listed in [Appendix E](#version-3).
Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities:
+* Support for an [adaptive metadata tree](#manifests), enabling efficient small commits, column updates, and columnar statistics representation
+* Data files and their corresponding deletion vector are stored in a combined entry, removing the need for two-phase planning
* Support for [relative locations](#file-locations-in-metadata) in metadata fields
* Writing new [equality deletes](#equality-delete-files) is no longer allowed
@@ -84,9 +86,9 @@ This table format tracks individual data files in a table instead of directories
Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table.
-Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, and its metrics. The data in a snapshot is the union of all live files in its manifests; each live file may only appear once (see [Content file uniqueness](#content-file-uniqueness)). Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data files with any subset of a table and are not [...]
+Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, an optional colocated deletion vector (v4), and its metrics. The data in a snapshot is the union of all live files in its manifests; each live file may only appear once (see [Content file uniqueness](#content-file-uniqueness)). Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data [...]
-The manifests that make up a snapshot are stored in a manifest list file. Each manifest list stores metadata about manifests, including partition stats and data file counts. These stats are used to avoid reading manifests that are not required for an operation.
+In v1-v3, the manifests that make up a snapshot are stored in a manifest list file. Each manifest list stores metadata about manifests, including partition stats and data file counts. These stats are used to avoid reading manifests that are not required for an operation. Starting in v4, manifest lists are replaced by a single root manifest per snapshot, which can contain references to data files, data manifests, and delete manifests in a unified structure.
### Optimistic Concurrency
@@ -145,8 +147,12 @@ Version 4 of the Iceberg spec adds support for relative locations in metadata, e
* **Schema** -- Names and types of fields in a table.
* **Partition spec** -- A definition of how partition values are derived from data fields.
* **Snapshot** -- The state of a table at some point in time, including the set of all data files.
-* **Manifest list** -- A file that lists manifest files; one per snapshot.
-* **Manifest** -- A file that lists data or delete files; a subset of a snapshot.
+* **Snapshot root file** -- The per-snapshot file that tracks a snapshot's manifests; a manifest list (v1-v3) or a root manifest (v4+).
+* **Manifest list** -- (v1-v3 only) A file that lists manifest files; one per snapshot.
+* **Root manifest** -- (v4+) A manifest that can reference data files, data manifests, or delete manifests; one per snapshot.
+* **Leaf manifest** -- A data or delete manifest.
+* **Data manifest** -- A file that lists data files and, in v4, their deletion vectors and column files; a subset of a snapshot.
+* **Delete manifest** -- A file that lists delete files to be associated with data files at planning time.
* **Data file** -- A file that contains rows of a table.
* **Delete file** -- A file that encodes rows of a table that are deleted by position or data values.
* **Absolute path** -- A path string that includes a [URI](https://datatracker.ietf.org/doc/html/rfc3986#section-3.1) scheme and can be used directly.
@@ -477,7 +483,7 @@ A data file with only new rows for the table may omit the `_last_updated_sequenc
On read, if `_last_updated_sequence_number` is `null` it is assigned the `sequence_number` of the data file's manifest entry. The data sequence number of a data file is documented in [Sequence Number Inheritance](#sequence-number-inheritance).
-When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its containing data file plus the row position in that data file (`_pos`). A data file's `first_row_id` field is assigned using inheritance and is documented in [First Row ID Inheritance](#first-row-id-inheritance). A manifest's `first_row_id` is assigned when writing the manifest list for a snapshot and is documented in [First Row ID Assignment](#first-row-id-assignment). A snapshot's `first-row-id` is set to th [...]
+When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its containing data file plus the row position in that data file (`_pos`). A data file's `first_row_id` field is assigned using inheritance and is documented in [First Row ID Inheritance](#first-row-id-inheritance). A manifest's `first_row_id` is assigned when writing the snapshot root file and is documented in [Snapshot Root File First Row ID Assignment](#snapshot-root-file-first-row-id-assignment). Starting in [...]
When an existing row is moved to a different data file for any reason, writers should write `_row_id` and `_last_updated_sequence_number` according to the following rules:
@@ -547,7 +553,7 @@ Note that:
### Partitioning
-Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. (Manifests store data files from any partition, as long as the partition spec is the same for the data files.)
+Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every ent [...]
Tables are configured with a **partition spec** that defines how to produce a tuple of partition values from a record. A partition spec has a list of fields that consist of:
@@ -657,15 +663,27 @@ A data or delete file is associated with a sort order by the sort order's id wit
### Manifests
-A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information. One or more manifest files are used to store a [snapshot](#snapshots), which tracks all of the files in a table at some point in time. Manifests are tracked by a [manifest list](#manifest-lists) for each table snapshot.
+A table's metadata tree is composed of manifests. A manifest is an immutable file that tracks a subset of a table metadata for a given [snapshot](#snapshots). Leaf manifests are the lowest level of the metadata tree and track data or delete files, along with each file's partition data, metrics, and tracking information. The snapshot root file tracks leaf manifests; in v4 and later the root is also a manifest and can also track data files.
A manifest is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection.
-A manifest may store either data files or delete files, but not both because manifests that contain delete files are scanned first during job planning. Whether a manifest is a data manifest or a delete manifest is stored in manifest metadata.
+Each manifest type contains the following content:
-A manifest stores files for a single partition spec. When a table’s partition spec changes, old files remain in the older manifest and newer files are written to a new manifest. This is required because a manifest file’s schema is based on its partition spec (see below). The partition spec of each manifest is also used to transform predicates on the table's data rows into predicates on partition values that are used during job planning to select files from a manifest.
+| Version | Manifest type | Contents | File format |
+|---------|-----------------|-------------------------------------------------|-------------|
+| v1-v3 | Data manifest | Data files | Avro |
+| v2-v3 | Delete manifest | Delete files | Avro |
+| v4 | Root manifest | Leaf manifests, data files, or v1-v3 manifests | Parquet |
+| v4 | Leaf manifest | Data files and their colocated deletion vectors | Parquet |
-A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
+- v1-v3: All entries in a manifest use the same partition spec, because the manifest's partition struct schema is derived from that spec.
+- v4: A manifest may store files written with different partition specs.
+
+The partition spec used when writing each data file is used to transform predicates on the table's data rows into predicates on partition values during job planning.
+
+#### Manifest File Format
+
+A manifest file must store metadata as properties in the file's key-value metadata:
=== "v1 - v3"
| v1 | v2 and v3 | Key | Value |
@@ -677,15 +695,26 @@ A manifest file must store the partition spec and other metadata as properties i
| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
+=== "v4"
+ | Requirement | Key | Value |
+ |-------------|---------------------|-------|
+ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
+ | _required_ | `format-version` | Table format version number of the manifest as a string |
+
#### Content file uniqueness
Within a snapshot, each content file must be referenced by at most one live manifest entry across all manifests; otherwise, the snapshot has undefined behavior. Writers should not produce multiple manifest entries for the same content file in a snapshot (for example, both ADDED and DELETED entries for the same file). Writers are not required to validate uniqueness at commit time.
-#### Manifest Entry Fields
+#### Manifest Schema
+
+Starting in v4, manifests store `tracked_file` records that describe data or manifest files and contain a `tracking` struct with metadata such as `status`. In v1-v3 manifests, this relationship is inverted: each `manifest_entry` contains tracking metadata and a `data_file` struct describing the file. The v4 manifest schema is not compatible with the v1-v3 manifest schema.
-The schema of a manifest file is defined by the `manifest_entry` struct, which consists of the following fields:
+The contents of a file are part of a snapshot if its tracking `status` is **live**: ADDED, EXISTING, or MODIFIED. Changes in a snapshot are tracked using the additional DELETED and REPLACED statuses.
=== "v1 - v3"
+ <a id="manifest-entry-fields"></a>
+ The v1-v3 `manifest_entry` struct has the following fields:
+
| v1 | v2 and v3 | Field id, name | Type | Description |
| ---------- | ---------- |-------------------------------|-----------------------------------------------------------|-------------|
| _required_ | _required_ | **`0 status`** | `int` with meaning: `0: EXISTING` `1: ADDED` `2: DELETED` | Used to track additions and deletions. Deletes are informational only and not used in scans. |
@@ -694,27 +723,15 @@ The schema of a manifest file is defined by the `manifest_entry` struct, which c
| | _optional_ | **`4 file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null and status is 1 (added). |
| _required_ | _required_ | **`2 data_file`** | `data_file` `struct` (see below) | File path, partition tuple, metrics, ... |
-The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-
-When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-
-When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted). The file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that older snapshots have also been garbage collected [1].
-
-Iceberg v2 adds data and file sequence numbers to the entry and makes the snapshot ID optional. Values for these fields are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the manifest list.
-The `sequence_number` field represents the data sequence number and must never change after a file is added to the dataset. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
-The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
-The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly.
-
-Notes:
+ The manifest entry fields are used to keep track of the snapshot in which files were added or logically deleted. The `data_file` struct, defined below, is nested inside the manifest entry so that it can be easily passed to job planning without the manifest entry fields.
-1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots.
-2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available.
+ When a file is added to the dataset, its manifest entry should store the snapshot ID in which the file was added and set status to 1 (added).
-##### Data File Fields
+ When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted).
-The `data_file` struct consists of the following fields:
+ <a id="data-file-fields"></a>
+ The `data_file` struct consists of the following fields:
-=== "v1 - v3"
| v1 | v2 | v3 | Field id, name | Type | Description |
| ---------- |------------|------------|-----------------------------------|-----------------------------------------------------------------------------|-------------|
| | _required_ | _required_ | **`134 content`** | `int` with meaning: `0: DATA`, `1: POSITION DELETES`, `2: EQUALITY DELETES` | Type of content stored by the data file: data, equality deletes, or position deletes (all v1 files are data files) |
@@ -723,7 +740,6 @@ The `data_file` struct consists of the following fields:
| _required_ | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids |
| _required_ | _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file, or the cardinality of a deletion vector |
| _required_ | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes |
- | | | | **`146 content_stats`** | `content_stats` `struct` | Container struct for per-field metrics structs. See [Content Stats](#content-stats) |
| _required_ | | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2 or v3.** |
| _optional_ | | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** |
| _optional_ | | | ~~**`107 sort_columns`**~~ | `list<112: int>` | **Deprecated. Do not write.** |
@@ -732,35 +748,141 @@ The `data_file` struct consists of the following fields:
| _optional_ | _optional_ | _optional_ | **`110 null_value_counts`** | `map<121: int, 122: long>` | Map from column id to number of null values in the column |
| _optional_ | _optional_ | _optional_ | **`137 nan_value_counts`** | `map<138: int, 139: long>` | Map from column id to number of NaN values in the column |
| _optional_ | _optional_ | | ~~**`111 distinct_counts`**~~ | `map<123: int, 124: long>` | **Deprecated. Do not write.** |
- | _optional_ | _optional_ | _optional_ | **`125 lower_bounds`** | `map<126: int, 127: binary>` | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] |
- | _optional_ | _optional_ | _optional_ | **`128 upper_bounds`** | `map<129: int, 130: binary>` | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] |
+ | _optional_ | _optional_ | _optional_ | **`125 lower_bounds`** | `map<126: int, 127: binary>` | Map from column id to lower bound in the column serialized as binary. Each value must be less than or equal to all non-null, non-NaN values in the column for the file. See [Field-level Metrics and Statistics](#field-level-metrics-and-statistics) |
+ | _optional_ | _optional_ | _optional_ | **`128 upper_bounds`** | `map<129: int, 130: binary>` | Map from column id to upper bound in the column serialized as binary. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file. See [Field-level Metrics and Statistics](#field-level-metrics-and-statistics) |
| _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption |
| _optional_ | _optional_ | _optional_ | **`132 split_offsets`** | `list<133: long>` | Split offsets for the data file. For example, all row group offsets in a Parquet file. Must be sorted ascending |
| | _optional_ | _optional_ | **`135 equality_ids`** | `list<136: int>` | Field ids used to determine row equality in equality delete files. Required when `content=2` and should be null otherwise. Fields with ids listed in this column must be present in the delete file |
- | _optional_ | _optional_ | _optional_ | **`140 sort_order_id`** | `int` | ID representing sort order for this file [3]. |
+ | _optional_ | _optional_ | _optional_ | **`140 sort_order_id`** | `int` | ID representing sort order for this file [1]. |
| | | _optional_ | **`142 first_row_id`** | `long` | The `_row_id` for the first row in the data file. See [First Row ID Inheritance](#first-row-id-inheritance) |
- | | _optional_ | _optional_ | **`143 referenced_data_file`** | `string` | Fully qualified location (URI with FS scheme) of a data file that all deletes reference [4] |
- | | | _optional_ | **`144 content_offset`** | `long` | The offset in the file where the content starts [5] |
- | | | _optional_ | **`145 content_size_in_bytes`** | `long` | The length of a referenced content stored in the file; required if `content_offset` is present [5] |
+ | | _optional_ | _optional_ | **`143 referenced_data_file`** | `string` | Fully qualified location (URI with FS scheme) of a data file that all deletes reference [2] |
+ | | | _optional_ | **`144 content_offset`** | `long` | The offset in the file where the content starts [3] |
+ | | | _optional_ | **`145 content_size_in_bytes`** | `long` | The length of a referenced content stored in the file; required if `content_offset` is present [3] |
-The `partition` struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct's field ids must match the ids from the partition spec.
+ The `partition` struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct's field ids must match the ids from the partition spec.
-The v4 `content_stats` container struct stores field-level metrics. Unlike the metrics maps, the type of `content_stats` is based on table metadata, like schema. Similar to the `partition` struct, the same type is used for all files tracked in a manifest.
+ Notes:
-Notes:
+ 1. If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files should be written with a non-null order id. [Position deletes](#position-delete-files) are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files.
+ 2. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a single data file. Setting the referenced file is required for deletion vectors.
+ 3. The `content_offset` and `content_size_in_bytes` fields are used to reference a specific blob for direct access to a deletion vector. For deletion vectors, these values are required and must exactly match the `offset` and `length` stored in the Puffin footer for the deletion vector blob.
+ 4. The following field ids are reserved on `data_file`: 141.
-1. Single-value serialization for lower and upper bounds is detailed in Appendix D.
-2. For `float` and `double`, the value `-0.0` must precede `+0.0`, as in the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper bounds.
-3. If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files should be written with a non-null order id. [Position deletes](#position-delete-files) are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files.
-4. Position delete metadata can use `referenced_data_file` when all deletes tracked by the entry are in a single data file. Setting the referenced file is required for deletion vectors.
-5. The `content_offset` and `content_size_in_bytes` fields are used to reference a specific blob for direct access to a deletion vector. For deletion vectors, these values are required and must exactly match the `offset` and `length` stored in the Puffin footer for the deletion vector blob.
-6. The following field ids are reserved on `data_file`: 141.
+=== "v4"
+ The `tracked_file` struct has the following fields:
+
+ | On write | Field id | Name | Type | Description |
+ |------------|----------|--------------------------|-------------------------------------------------------|-------------|
+ | _required_ | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | Type of content stored in the entry. |
+ | _required_ | 147 | **`tracking`** | `tracking` struct | Tracking metadata like status, snapshot ID, and sequence number. See [Tracking](#tracking-fields). |
+ | _required_ | 100 | **`location`** | `string` | Location of the file. |
+ | _required_ | 101 | **`file_format`** | `string` | String file format name: `avro`, `orc`, or `parquet` |
+ | _required_ | 104 | **`file_size_in_bytes`** | `long` | Total file size in bytes. |
+ | _required_ | 103 | **`record_count`** | `long` | Number of records in this file. |
+ | _optional_ | 131 | **`key_metadata`** | `binary` | Key metadata for encryption; specific to the encryption scheme. |
+ | _optional_ | 132 | **`split_offsets`** | `list<133: long>` | Split offsets for the data file. Must be sorted ascending. |
+ | _optional_ | 141 | **`spec_id`** | `int` | ID of the partition spec used to partition the file; null if unpartitioned |
+ | _optional_ | 102 | **`partition`** | `struct<...>` | Partition data tuple for the file; null if unpartitioned. |
+ | _optional_ | 140 | **`sort_order_id`** | `int` | ID representing sort order for this file. If missing or unknown, the order is assumed to be unsorted. |
+ | _optional_ | 146 | **`content_stats`** | `content_stats` struct | Field-level stats. See [Content Stats](#content-stats). |
+ | _optional_ | 150 | **`manifest_info`** | `manifest_info` struct | Manifest-specific stats. See [Manifest Info](#manifest-info-fields). |
+ | _optional_ | 148 | **`deletion_vector`** | `deletion_vector` struct | Row-level deletion vector for a data file. See [Deletion Vector](#deletion-vector-fields). |
+ | _optional_ | 158 | **`column_files`** | `list<159: column_file>` | Column files associated with this file. See [Column File](#column-file-fields). |
+
+ <a id="tracking-fields"></a>
+ The `tracking` struct has the following fields:
+
+ | On write | Field id | Name | Type | Description |
+ |------------|----------|-------------------------------|---------------------------------------------------------------------|-------------|
+ | _required_ | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3: REPLACED, 4: MODIFIED) | Used to track additions, deletions, replacements, and modifications. |
+ | _optional_ | 1 | **`snapshot_id`** | `long` | Snapshot ID where the file was added, replaced, or deleted. Inherited when null. |
+ | _optional_ | 5 | **`modified_snapshot_id`** | `long` | Snapshot ID where the file was last modified. |
+ | _optional_ | 3 | **`sequence_number`** | `long` | Data sequence number of the file. Inherited when null. See [Sequence Number Inheritance](#sequence-number-inheritance). |
+ | _optional_ | 4 | **`file_sequence_number`** | `long` | File sequence number indicating when the file was added. Inherited when null. See [Sequence Number Inheritance](#sequence-number-inheritance). |
+ | _optional_ | 142 | **`first_row_id`** | `long` | Base row ID for assigning `_row_id` values. See [First Row ID Inheritance](#first-row-id-inheritance). |
+ | _optional_ | 6 | **`deleted_positions`** | `binary` | Positions deleted via manifest DV in the `modified_snapshot_id` snapshot. See [Manifest Deletion Vectors](#manifest-deletion-vectors). |
+ | _optional_ | 7 | **`replaced_positions`** | `binary` | Positions replaced via manifest DV in the `modified_snapshot_id` snapshot. See [Manifest Deletion Vectors](#manifest-deletion-vectors). |
+
+ <a id="deletion-vector-fields"></a>
+ The `deletion_vector` struct has the following fields:
+
+ | On write | Field id | Name | Type | Description |
+ |------------|----------|---------------------|----------|-------------|
+ | _required_ | 155 | **`location`** | `string` | Location of the file that stores the DV. |
+ | _required_ | 144 | **`offset`** | `long` | Offset in the file where the content starts. |
+ | _required_ | 145 | **`size_in_bytes`** | `long` | Length of the referenced content stored in the file. |
+ | _required_ | 156 | **`cardinality`** | `long` | Number of set bits (deleted rows) in the deletion vector. |
+ | _optional_ | 149 | **`key_metadata`** | `binary` | Key metadata for encryption; specific to the encryption scheme. |
+
+ <a id="manifest-info-fields"></a>
+ The `manifest_info` struct has the following fields:
+
+ | On write | Field id | Name | Type | Description |
+ |------------|----------|----------------------------|--------------------------|-------------|
+ | _required_ | 521 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | Format version used to write the manifest. |
+ | _optional_ | 522 | **`dv`** | `binary` | Positions in the referenced leaf manifest that are not live. See [Manifest Deletion Vectors](#manifest-deletion-vectors). |
+ | _required_ | 504 | **`added_files_count`** | `int` | Count of entries with status ADDED in the manifest. |
+ | _required_ | 505 | **`existing_files_count`** | `int` | Count of entries with status EXISTING in the manifest. |
+ | _required_ | 525 | **`modified_files_count`** | `int` | Count of entries with status MODIFIED in the manifest. |
+ | _required_ | 506 | **`deleted_files_count`** | `int` | Count of entries with status DELETED in the manifest. |
+ | _required_ | 523 | **`replaced_files_count`** | `int` | Count of entries with status REPLACED in the manifest. |
+ | _required_ | 512 | **`added_rows_count`** | `long` | Total number of rows in ADDED entries. |
+ | _required_ | 513 | **`existing_rows_count`** | `long` | Total number of rows in EXISTING entries. |
+ | _required_ | 526 | **`modified_rows_count`** | `long` | Total number of rows in MODIFIED entries. |
+ | _required_ | 514 | **`deleted_rows_count`** | `long` | Total number of rows in DELETED entries. |
+ | _required_ | 524 | **`replaced_rows_count`** | `long` | Total number of rows in REPLACED entries. |
+ | _required_ | 516 | **`min_sequence_number`** | `long` | Minimum data sequence number of all live entries in the manifest. |
+
+ <a id="column-file-fields"></a>
+ The `column_file` struct has the following fields:
+
+ | On write | Field id | Name | Type | Description |
+ |------------|----------|--------------------------|------------------|-------------|
+ | _required_ | 160 | **`location`** | `string` | Location of the column file. |
+ | _required_ | 161 | **`field_ids`** | `list<162: int>` | Live field IDs stored in this column file. |
+ | _required_ | 163 | **`file_format`** | `string` | String file format name: `avro`, `orc`, or `parquet`. |
+ | _required_ | 164 | **`file_size_in_bytes`** | `long` | Total column file size in bytes. |
+ | _optional_ | 165 | **`key_metadata`** | `binary` | Key metadata for encryption; specific to the encryption scheme. |
+
+ <a id="tracked-file-requirements"></a>
+ Tracked files must meet the following requirements:
+
+ - Manifest contents:
+ - A leaf manifest written in v4 may only contain data files.
+ - A root manifest may contain manifests from any format version, but only v4 leaf manifests may be created by writers.
+ - Delete files from pre-v4 tables are valid in upgraded tables and are tracked in delete manifests written before the upgrade.
+ - `tracking`:
+ - `tracking.snapshot_id`, `tracking.sequence_number`, and `tracking.file_sequence_number` are required for all tracked files in the root manifest. `tracking.first_row_id` is also required for data files, data manifests, and v4 leaf manifests in the root manifest.
+ - Writers should not write a null `tracking.snapshot_id`.
+ - `tracking.deleted_positions` and `tracking.replaced_positions` may only be set if the tracked file is a manifest.
+ - `spec_id`:
+ - For manifests, `spec_id` must be set to the `spec_id` of the manifest's entries if all entries have the same `spec_id`, and must be null otherwise.
+ - `manifest_info`:
+ - `manifest_info` must be set if and only if the tracked file is a manifest.
+ - `manifest_info.format_version` must be V4 for manifests written in v4 and PRE-V4 for manifests written by earlier versions.
+ - For manifests, `manifest_info.added_files_count`, `existing_files_count`, `deleted_files_count`, `replaced_files_count`, and `modified_files_count` must sum to `record_count`.
+ - `deletion_vector`:
+ - `deletion_vector` may only be set if the tracked file is a data file.
+ - Row-level deletes may only be written in v4 as deletion vectors in the data file's `deletion_vector`.
+ - `deletion_vector.offset` and `deletion_vector.size_in_bytes` must exactly match the `offset` and `length` stored in the Puffin footer for the deletion vector blob.
+ - `column_files`:
+ - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest.
+ - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`.
+
+ <a id="updating-tracking-metadata"></a>
+ When a file is added to the dataset, its tracked file must set status to ADDED and store the snapshot ID in which the file was added.
+
+ When a data file's deletion vector or column files are updated, the writer must produce two entries: a MODIFIED entry for the updated live version and a REPLACED entry with the previous metadata (for change detection). The MODIFIED entry's `modified_snapshot_id` must record the snapshot ID in which the change occurred. The REPLACED entry can be produced in place by setting its position in the leaf manifest's [`manifest_info.dv`](#manifest-deletion-vectors) (to remove it from planning [...]
+
+ When a file is deleted from the dataset, the deletion must be recorded in the snapshot that deletes the file with a DELETED entry that sets `snapshot_id` to the snapshot in which the file was deleted. The DELETED entry can be produced by setting its position in the leaf manifest's `tracking.deleted_positions` and [`manifest_info.dv`](#manifest-deletion-vectors) and updating the leaf manifest's `tracking.modified_snapshot_id` to the new snapshot ID.
+
+ A leaf manifest whose `manifest_info.dv` changed must have status MODIFIED and record the snapshot in its `modified_snapshot_id`. When new positions are set in `manifest_info.dv`, those positions must be set in either `tracking.deleted_positions` or `tracking.replaced_positions` to record what the entry's state changed to in the snapshot identified by `modified_snapshot_id`. Deleted and replaced bitmaps may be removed when the manifest metadata is copied to a new root.
##### Field-level Metrics and Statistics
Field statistics (or, interchangeably, metrics) are used when filtering to select data and delete files.
-In v3 and earlier, metrics are stored in maps keyed by field id: `value_counts`, `null_value_counts`, `nan_value_counts`, `lower_bounds` and `upper_bounds`.
+In v3 and earlier, metrics are stored in maps keyed by field id: `value_counts`, `null_value_counts`, `nan_value_counts`, `lower_bounds` and `upper_bounds`. Bounds in these maps use the single-value serialization detailed in [Appendix D](#appendix-d-single-value-serialization).
In v4, metrics are stored as typed values in the `content_stats` struct, documented in the [Content Stats](#content-stats) section.
@@ -768,6 +890,10 @@ Both representations store equivalent information. If a map or id in a map is mi
For delete files, metrics must store bounds and counts for all deleted rows, or must be omitted. Storing metrics for deleted rows ensures that the values can be used during job planning to find delete files that must be merged during a scan.
+###### Bounds for Float and Double
+
+For `float` and `double`, the value `-0.0` must precede `+0.0`, as in the IEEE 754 `totalOrder` predicate. NaNs are not permitted as lower or upper bounds.
+
###### Bounds for Geometry and Geography
For `geometry` and `geography` types, `lower_bounds` and `upper_bounds` are points with coordinates X, Y, Z (optional), and M (optional) that represent the lower and upper bounds of a bounding box that contains all objects in the file.
@@ -925,26 +1051,33 @@ Fields with stats tracked in `content_stats` change based on updates like schema
A simple (and recommended) way for writers to adapt existing metadata for table changes is to read manifests with the implementation's current `content_stats` type and apply schema evolution rules, such as reading `int` as `long` for promoted fields.
-#### Sequence Number Inheritance
+#### Inheritance
+
+Values for `snapshot_id`, `sequence_number`, and `file_sequence_number` are inherited from manifest metadata when `null`. That is, if the field is `null` for an entry, then the entry must inherit its value from the manifest file's metadata, stored in the snapshot root file.
+
+##### Sequence Number Inheritance
Manifests track the sequence number when a data or delete file was added to the table.
-When adding a new file, its data and file sequence numbers are set to `null` because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing `null` with the manifest's sequence number from the manifest list.
+The `sequence_number` field represents the data sequence number. It is set when a file is added to the dataset and must not change unless a column file is added. The data sequence number represents a relative age of the file content and should be used for planning which delete files apply to a data file.
+The `file_sequence_number` field represents the sequence number of the snapshot that added the file and must also remain unchanged upon assigning at commit. The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number.
+
+When adding a new file to a leaf manifest, its data and file sequence numbers are set to `null` because the snapshot's sequence number is not assigned until the snapshot is successfully committed. When reading, sequence numbers are inherited by replacing `null` with the manifest's sequence number from the snapshot root file.
It is also possible to add a new file with data that logically belongs to an older sequence number. In that case, the data sequence number must be provided explicitly and not inherited. However, the file sequence number must be always assigned when the snapshot is successfully committed.
-When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time.
+The data sequence number is inherited only if the entry status is ADDED, or MODIFIED by adding a column file. The file sequence number is inherited only if the entry status is ADDED. When writing an existing file to a new manifest or marking an existing file as deleted, the data and file sequence numbers must be non-null and set to the original values that were either inherited or provided at the commit time.
-Inheriting sequence numbers through the metadata tree allows writing a new manifest without a known sequence number, so that a manifest can be written once and reused in commit retries. To change a sequence number for a retry, only the manifest list must be rewritten.
+Inheriting sequence numbers through the metadata tree allows writing a new manifest without a known sequence number, so that a manifest can be written once and reused in commit retries. To change a sequence number for a retry, only the snapshot root file must be rewritten.
-When reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0.
+Inheritance does not apply to v1, which does not have inherited fields like sequence number. When reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0.
-#### First Row ID Inheritance
+##### First Row ID Inheritance
-When adding a new data file, its `first_row_id` field is set to `null` because it is not assigned until the snapshot is successfully committed.
+When adding a new data file to a leaf manifest, its `first_row_id` field is set to `null` because it is not assigned until the snapshot is successfully committed.
When reading, the `first_row_id` is assigned by replacing `null` with the manifest's `first_row_id` plus the sum of `record_count` for all data files that preceded the file in the manifest that also had a null `first_row_id`.
-The inherited value of `first_row_id` must be written into data file metadata when creating existing and deleted entries. The value of `first_row_id` for delete files is always `null`.
+The inherited value of `first_row_id` must be written into data file metadata for all entries with a status other than `ADDED`. The value of `first_row_id` for delete files is always `null`.
Any null (unassigned) `first_row_id` must be assigned via inheritance, even if the data file is existing. This ensures that row IDs are assigned to existing data files in upgraded tables in the first commit after upgrading to v3.
@@ -966,6 +1099,20 @@ A snapshot consists of the following fields:
| | | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
+=== "v4"
+ | v4 | Field | Description |
+ | ---------- |------------------------------|-------------|
+ | _required_ | **`snapshot-id`** | A unique long ID |
+ | _optional_ | **`parent-snapshot-id`** | The snapshot ID of the snapshot's parent. Omitted for any snapshot with no parent |
+ | _required_ | **`sequence-number`** | A monotonically increasing long that tracks the order of changes to a table |
+ | _required_ | **`timestamp-ms`** | A timestamp when the snapshot was created, used for garbage collection and table inspection |
+ | _required_ | **`root-manifest`** | The location of the root manifest for this snapshot |
+ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) |
+ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created |
+ | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) |
+ | _required_ | **`added-rows`** | The upper bound of the number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
+ | _optional_ | **`key-id`** | ID of the encryption key that encrypts the root manifest key metadata |
+
When committed while the table is at version 4 or higher, a snapshot's `timestamp-ms` must be strictly greater than its parent snapshot's `timestamp-ms` on the same branch. Snapshots with no parent, and snapshots that were committed before a table was upgraded to v4 or higher, do not have this requirement.
The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are:
@@ -980,10 +1127,10 @@ For other optional snapshot summary fields, see [Appendix F](#optional-snapshot-
Data and delete files for a snapshot can be stored in more than one manifest. This enables:
* Appends can add a new manifest to minimize the amount of data written, instead of adding new records by rewriting and appending to an existing manifest. (This is called a “fast append”.)
-* Tables can use multiple partition specs. A table’s partition configuration can evolve if, for example, its data volume changes. Each manifest uses a single partition spec, and queries do not need to change because partition filters are derived from data predicates.
+* Tables can use multiple partition specs. A table’s partition configuration can evolve if, for example, its data volume changes. Partition predicates for a partition spec are derived from data predicates and can be applied to filter files written using that spec. Prior to v4, a manifest stored files partitioned by a single spec.
* Large tables can be split across multiple manifests so that implementations can parallelize job planning or reduce the cost of rewriting a manifest.
-Manifests for a snapshot are tracked by a manifest list.
+Manifests for a snapshot are tracked by the snapshot root file and are not allowed in leaf manifest files.
Valid snapshots are stored as a list in table metadata. For serialization, see Appendix C.
@@ -991,16 +1138,28 @@ Valid snapshots are stored as a list in table metadata. For serialization, see A
A snapshot's `first-row-id` is assigned to the table's current `next-row-id` on each commit attempt. If a commit is retried, the `first-row-id` must be reassigned based on the table's current `next-row-id`. The `first-row-id` field is required even if a commit does not assign any ID space.
-The snapshot's `first-row-id` is the starting `first_row_id` assigned to manifests in the snapshot's manifest list.
+The snapshot's `first-row-id` is the starting `first_row_id` assigned to manifests in the snapshot root file. In v4, this includes data files in the root manifest.
The snapshot's `added-rows` captures the upper bound of the number of rows with assigned row IDs.
It can be used safely to increment the table's `next-row-id` during a commit.
It can be more than the number of rows added in this snapshot and include some existing rows,
see [Row Lineage Example](#row-lineage-example).
+<a id="first-row-id-assignment"></a>
+#### Snapshot Root File First Row ID Assignment
+
+The `first_row_id` for existing manifests must be preserved when writing a new snapshot root file. The value of `first_row_id` for delete manifests is always `null`. The `first_row_id` is only assigned for data manifests that do not have a `first_row_id`. Assignment must account for data files that will be assigned `first_row_id` values when the manifest is read. Data files in the root manifest (v4+) must also have a `first_row_id`: existing values must be preserved and `first_row_id` is [...]
+
+The first file in the snapshot root file without a `first_row_id` is assigned a value that is greater than or equal to the `first_row_id` of the snapshot. Subsequent files without a `first_row_id` are assigned one based on the previous file to be assigned a `first_row_id`. Each assigned `first_row_id` must be greater than or equal to the last assigned `first_row_id` plus the row count of the last assigned file, where the row count is:
+
+* For a manifest, the sum of `record_count` for its data files with a null `first_row_id`.
+* For a data file, its `record_count`.
+
+A simple and valid approach is to estimate the number of rows in data files that will be assigned a `first_row_id` using the manifest's `added_rows_count` and `existing_rows_count`: `first_row_id = last_assigned.first_row_id + last_assigned.added_rows_count + last_assigned.existing_rows_count`.
+
### Manifest Lists
-Snapshots are embedded in table metadata, but the list of manifests for a snapshot are stored in a separate manifest list file.
+Snapshots are embedded in table metadata, but the manifests for a snapshot are tracked in a separate snapshot root file. Manifest lists are the snapshot root file used by v1-v3; v4 uses a [root manifest](#manifests).
A new manifest list is written for each attempt to commit a snapshot because the list of manifests always changes to produce a new snapshot. When a manifest list is written, the (optimistic) sequence number of the snapshot is written for all new manifest files tracked by the list.
@@ -1028,7 +1187,7 @@ Manifest list files store `manifest_file`, a struct with the following fields:
| _optional_ | _required_ | _required_ | **`514 deleted_rows_count`** | `long` | Number of rows in all of files in the manifest that have status `DELETED`, when `null` this is assumed to be non-zero |
| _optional_ | _optional_ | _optional_ | **`507 partitions`** | `list<508: field_summary>` **(see below)** | A list of field summaries for each partition field in the spec. Each field in the list corresponds to a field in the manifest file’s partition spec. |
| _optional_ | _optional_ | _optional_ | **`519 key_metadata`** | `binary` | Implementation-specific key metadata for encryption |
- | | | _optional_ | **`520 first_row_id`** | `long` | The starting `_row_id` to assign to rows added by `ADDED` data files [First Row ID Assignment](#first-row-id-assignment) |
+ | | | _optional_ | **`520 first_row_id`** | `long` | The starting `_row_id` to assign to rows added by `ADDED` data files [Snapshot Root File First Row ID Assignment](#snapshot-root-file-first-row-id-assignment) |
`field_summary` is a struct with the following fields:
=== "v1 - v3"
@@ -1044,37 +1203,32 @@ Notes:
1. Lower and upper bounds are serialized to bytes using the single-object serialization in Appendix D. The type of used to encode the value is the type of the partition field data.
2. If -0.0 is a value of the partition field, the `lower_bound` must not be +0.0, and if +0.0 is a value of the partition field, the `upper_bound` must not be -0.0.
-#### First Row ID Assignment
+### Scan Planning
-The `first_row_id` for existing manifests must be preserved when writing a new manifest list. The value of `first_row_id` for delete manifests is always `null`. The `first_row_id` is only assigned for data manifests that do not have a `first_row_id`. Assignment must account for data files that will be assigned `first_row_id` values when the manifest is read.
+A reader plans a scan by producing live data files from the snapshot root file and any live leaf manifests referenced by the root.
-The first manifest without a `first_row_id` is assigned a value that is greater than or equal to the `first_row_id` of the snapshot. Subsequent manifests without a `first_row_id` are assigned one based on the previous manifest to be assigned a `first_row_id`. Each assigned `first_row_id` must increase by the row count of all files that will be assigned a `first_row_id` via inheritance in the last assigned manifest. That is, each `first_row_id` must be greater than or equal to the last as [...]
+A scan uses only [live](#manifest-schema) entries.
-A simple and valid approach is to estimate the number of rows in data files that will be assigned a `first_row_id` using the manifest's `added_rows_count` and `existing_rows_count`: `first_row_id = last_assigned.first_row_id + last_assigned.added_rows_count + last_assigned.existing_rows_count`.
+Manifests that contain no matching files, determined using file counts, partition summaries, or column stats, may be skipped.
-### Scan Planning
+Using content stats, a manifest is filtered by evaluating scan predicates against the column bounds and counts of each tracked file. The same filter logic can be used for both data and delete files because both store metrics of the rows either inserted or deleted. If metrics show that a delete file has no rows that match a scan predicate, it may be ignored just as a data file would be ignored [1].
-Scans are planned by reading the manifest files for the current snapshot. Deleted entries in data and delete manifests (those marked with status "DELETED") are not used in a scan.
+v1-v3 manifests are filtered using scan predicates converted to partition predicates, which filter partition tuples. These partition predicates are used to select relevant data files, delete files, and deletion vector metadata. Conversion uses the partition spec that was used to write the manifest file regardless of the current partition spec. Column bounds and counts stored by field id in metrics maps are used in the same way as content stats.
-Manifests that contain no matching files, determined using either file counts or partition summaries, may be skipped.
-
-For each manifest, scan predicates, which filter data rows, are converted to partition predicates, which filter partition tuples. These partition predicates are used to select relevant data files, delete files, and deletion vector metadata. Conversion uses the partition spec that was used to write the manifest file regardless of the current partition spec.
-
-Scan predicates are converted to partition predicates using an _inclusive projection_: if a scan predicate matches a row, then the partition predicate must match that row’s partition. This is called _inclusive_ [1] because rows that do not match the scan predicate may be included in the scan by the partition predicate.
+Scan predicates are converted to partition predicates using an _inclusive projection_: if a scan predicate matches a row, then the partition predicate must match that row’s partition. This is called _inclusive_ [2] because rows that do not match the scan predicate may be included in the scan by the partition predicate.
For example, an `events` table with a timestamp column named `ts` that is partitioned by `ts_day=day(ts)` is queried by users with ranges over the timestamp column: `ts > X`. The inclusive projection is `ts_day >= day(X)`, which is used to select files that may have matching rows. Note that, in most cases, timestamps just before `X` will be included in the scan because the file contains rows that match the predicate and rows that do not match the predicate.
The inclusive projection for an unknown partition transform is _true_ because the partition field is ignored and not used in filtering.
-Scan predicates are also used to filter data and delete files using column bounds and counts that are stored by field id in manifests. The same filter logic can be used for both data and delete files because both store metrics of the rows either inserted or deleted. If metrics show that a delete file has no rows that match a scan predicate, it may be ignored just as a data file would be ignored [2].
-
Data files that match the query filter must be read by the scan.
Duplicate live manifest entries for the same content file violate [content file uniqueness](#content-file-uniqueness). When duplicate live entries are present, table behavior (including scan results) is undefined. Reader implementations may raise an error but are not required to do so.
Delete files and deletion vector metadata that match the filters must be applied to data files at read time, limited by the following scope rules.
-* A deletion vector must be applied to a data file when all of the following are true:
+* A colocated deletion vector must be applied to the data file tracked by the same entry. No path, sequence number, or partition comparison applies because the vector is colocated with the data file.
+* A deletion vector tracked in a delete manifest must be applied to a data file when all of the following are true:
- The data file's `file_path` is equal to the deletion vector's `referenced_data_file`
- The data file's data sequence number is _less than or equal to_ the deletion vector's data sequence number
- The data file's partition (both spec and partition values) is equal [4] to the deletion vector's partition
@@ -1094,8 +1248,8 @@ In general, deletes are applied only to data files that are older and in the sam
Notes:
-1. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan.
-2. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected.
+1. For example, if `file_a` has rows with `id` between 1 and 10 and a delete file contains rows with `id` between 1 and 4, a scan for `id = 9` may ignore the delete file because none of the deletes can match a row that will be selected.
+2. An alternative, *strict projection*, creates a partition predicate that will match a file if all of the rows in the file must match the scan predicate. These projections are used to calculate the residual predicates for each file in a scan.
3. Floating point partition values are considered equal if their IEEE 754 floating-point "single format" bit layout are equal with NaNs normalized to have only the most significant mantissa bit set (the equivalent of calling `Float.floatToIntBits` or `Double.doubleToLongBits` in Java). The Avro specification requires all floating point values to be encoded in this format.
4. Unknown partition transforms do not affect partition equality. Although partition fields with unknown transforms are ignored for filtering, the result of an unknown transform is still used when testing whether partition values are equal.
@@ -1133,6 +1287,14 @@ When expiring snapshots, retention policies in table and snapshot references are
2. The snapshot is not one of the first `min-snapshots-to-keep` in the branch (including the branch's referenced snapshot)
5. Expire any snapshot not in the set of snapshots to retain.
+#### Deleting Files
+
+A file may be deleted from the file system when the snapshot in which it was deleted is garbage collected, assuming that the file is not live in any remaining snapshot [1].
+
+Notes:
+
+1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. When all snapshots are ancestors of a single branch, it is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental dele [...]
+
### Table Metadata
Table metadata is stored as JSON. Each table metadata change creates a new table metadata file that is committed by an atomic operation. This operation is used to ensure that a new version of table metadata replaces the version on which it was based. This produces a linear history of table versions and ensures that concurrent writes are not lost.
@@ -1171,6 +1333,7 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | _optional_ | **`partition-statistics`** | A list (optional) of [partition statistics](#partition-statistics). |
| | | _required_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot’s `first-row-id`. See [Row Lineage](#row-lineage). |
| | | _optional_ | **`encryption-keys`** | A list (optional) of [encryption keys](#encryption-keys) used for table encryption. |
+
=== "v4"
| v4 | Field | Description |
|------------|-----------------------------|-------------|
@@ -1370,10 +1533,12 @@ When removing a data file, writers must also remove any deletion vector that app
Row-level delete files (both equality and position delete files) are valid Iceberg data files: files must use valid Iceberg formats, schemas, and column projection. It is recommended that these delete files are written using the table's default file format.
-Row-level delete files and deletion vectors are tracked by manifests. A separate set of manifests is used for delete files and DVs, but the same manifest schema is used for both data and delete manifests. Deletion vectors are tracked individually by file location, offset, and length within the containing file. Deletion vector metadata must include the referenced data file.
-
Both position and equality delete files allow encoding deleted row values with a delete. This can be used to reconstruct a stream of changes to a table.
+Row-level delete files and deletion vectors are tracked by manifests. Delete manifests (v2-v3) track delete files and DVs separately from data manifests, using the same manifest schema. Deletion vectors in delete manifests are tracked individually by file location, offset, and length within the containing file, and their metadata must include the referenced data file.
+
+Starting in v4, writers must colocate new deletion vectors with their data file using the [`deletion_vector`](#deletion-vector-fields) field. Deletion vectors in delete manifests written before an upgrade to v4 remain valid.
+
#### Deletion Vectors
Deletion vectors identify deleted rows of a file by encoding deleted positions in a bitmap. A set bit at position P indicates that the row at position P is deleted.
@@ -1390,6 +1555,20 @@ At most one deletion vector is allowed per data file in a snapshot. If a DV is w
[puffin-spec]: puffin-spec.md
+#### Manifest Deletion Vectors
+
+Manifest deletion vectors identify entries of a leaf manifest that are no longer live by encoding their positions in a bitmap. A set bit at position P indicates that the entry at position P in the leaf manifest was either deleted or replaced.
+
+Manifest deletion vectors are encoded using the [Mumbling bitmap spec][mumbling-spec] and stored inline on the root manifest entry that references the leaf manifest. The snapshot in which the vector last changed is recorded in `tracking.modified_snapshot_id`; the three bitmaps are:
+
+* `manifest_info.dv`: every position in the leaf manifest that is not live.
+* `tracking.deleted_positions`: the positions deleted in the `modified_snapshot_id` snapshot.
+* `tracking.replaced_positions`: the positions replaced in the `modified_snapshot_id` snapshot.
+
+`deleted_positions` and `replaced_positions` are disjoint.
+
+[mumbling-spec]: https://iceberg.apache.org/mumbling-spec/
+
#### Position Delete Files
Position-based delete files identify deleted rows by file and position in one or more data files, and may optionally contain the deleted row.