You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by ke...@apache.org on 2025/08/16 02:21:38 UTC
(iceberg-python) branch main updated: docs: clarify Parameters for the add_files API (#2249)
This is an automated email from the ASF dual-hosted git repository.
kevinjqliu pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-python.git
The following commit(s) were added to refs/heads/main by this push:
new 80135451 docs: clarify Parameters for the add_files API (#2249)
80135451 is described below
commit 80135451d030569259d83674ef147e0d6f62fd51
Author: Daniel Tse <xi...@gmail.com>
AuthorDate: Sat Aug 16 10:21:33 2025 +0800
docs: clarify Parameters for the add_files API (#2249)
## Summary
Related Issue: #2132
1. This PR enhances the documentation for the add_files API by:
2. Adding a parameter table to clarify the required and optional inputs
and outputs.
3. Providing a complete example that includes all parameters, such as
snapshot_properties and check_duplicate_files.
4. Strengthening the warning regarding the default setting of
check_duplicate_files=True and the associated risks of disabling it.
---------
Co-authored-by: “Zhibin <zh...@shopee.com>
---
mkdocs/docs/api.md | 59 ++++++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 51 insertions(+), 8 deletions(-)
diff --git a/mkdocs/docs/api.md b/mkdocs/docs/api.md
index 070a6d08..89d5692d 100644
--- a/mkdocs/docs/api.md
+++ b/mkdocs/docs/api.md
@@ -1004,6 +1004,33 @@ To show only data files or delete files in the current snapshot, use `table.insp
Expert Iceberg users may choose to commit existing parquet files to the Iceberg table as data files, without rewriting them.
+<!-- prettier-ignore-start -->
+
+!!! note "Name Mapping"
+ Because `add_files` uses existing files without writing new parquet files that are aware of the Iceberg's schema, it requires the Iceberg's table to have a [Name Mapping](https://iceberg.apache.org/spec/?h=name+mapping#name-mapping-serialization) (The Name mapping maps the field names within the parquet files to the Iceberg field IDs). Hence, `add_files` requires that there are no field IDs in the parquet file's metadata, and creates a new Name Mapping based on the table's current sc [...]
+
+!!! note "Partitions"
+ `add_files` only requires the client to read the existing parquet files' metadata footer to infer the partition value of each file. This implementation also supports adding files to Iceberg tables with partition transforms like `MonthTransform`, and `TruncateTransform` which preserve the order of the values after the transformation (Any Transform that has the `preserves_order` property set to True is supported). Please note that if the column statistics of the `PartitionField`'s sour [...]
+
+!!! warning "Maintenance Operations"
+ Because `add_files` commits the existing parquet files to the Iceberg Table as any other data file, destructive maintenance operations like expiring snapshots will remove them.
+
+!!! warning "Check Duplicate Files"
+ The `check_duplicate_files` parameter determines whether the method validates that the specified `file_paths` do not already exist in the Iceberg table. When set to True (the default), the method performs a validation against the table’s current data files to prevent accidental duplication, helping to maintain data consistency by ensuring the same file is not added multiple times. While this check is important for data integrity, it can introduce performance overhead for tables with [...]
+
+<!-- prettier-ignore-end -->
+
+### Usage
+
+| Parameter | Required? | Type | Description |
+| ------------------------- | --------- | -------------- | ----------------------------------------------------------------------- |
+| `file_paths` | ✔️ | List[str] | The list of full file paths to be added as data files to the table |
+| `snapshot_properties` | | Dict[str, str] | Properties to set for the new snapshot. Defaults to an empty dictionary |
+| `check_duplicate_files` | | bool | Whether to check for duplicate files. Defaults to `True` |
+
+### Example
+
+Add files to Iceberg table:
```python
# Given that these parquet files have schema consistent with the Iceberg table
@@ -1019,18 +1046,34 @@ tbl.add_files(file_paths=file_paths)
# A new snapshot is committed to the table with manifests pointing to the existing parquet files
```
-<!-- prettier-ignore-start -->
+Add files to Iceberg table with custom snapshot properties:
+```python
+# Assume an existing Iceberg table object `tbl`
-!!! note "Name Mapping"
- Because `add_files` uses existing files without writing new parquet files that are aware of the Iceberg's schema, it requires the Iceberg's table to have a [Name Mapping](https://iceberg.apache.org/spec/?h=name+mapping#name-mapping-serialization) (The Name mapping maps the field names within the parquet files to the Iceberg field IDs). Hence, `add_files` requires that there are no field IDs in the parquet file's metadata, and creates a new Name Mapping based on the table's current sc [...]
+file_paths = [
+ "s3a://warehouse/default/existing-1.parquet",
+ "s3a://warehouse/default/existing-2.parquet",
+]
-!!! note "Partitions"
- `add_files` only requires the client to read the existing parquet files' metadata footer to infer the partition value of each file. This implementation also supports adding files to Iceberg tables with partition transforms like `MonthTransform`, and `TruncateTransform` which preserve the order of the values after the transformation (Any Transform that has the `preserves_order` property set to True is supported). Please note that if the column statistics of the `PartitionField`'s sour [...]
+# Custom snapshot properties
+snapshot_properties = {"abc": "def"}
-!!! warning "Maintenance Operations"
- Because `add_files` commits the existing parquet files to the Iceberg Table as any other data file, destructive maintenance operations like expiring snapshots will remove them.
+# Enable duplicate file checking
+check_duplicate_files = True
-<!-- prettier-ignore-end -->
+# Add the Parquet files to the Iceberg table without rewriting
+tbl.add_files(
+ file_paths=file_paths,
+ snapshot_properties=snapshot_properties,
+ check_duplicate_files=check_duplicate_files
+)
+
+# NameMapping must have been set to enable reads
+assert tbl.name_mapping() is not None
+
+# Verify that the snapshot property was set correctly
+assert tbl.metadata.snapshots[-1].summary["abc"] == "def"
+```
## Schema evolution