You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by pv...@apache.org on 2026/09/07 10:05:02 UTC
(iceberg) branch main updated: Docs, Flink: Update Flink configuration and write options in Iceberg documentation (#17938)
This is an automated email from the ASF dual-hosted git repository.
pvary pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 141f1d0475 Docs, Flink: Update Flink configuration and write options in Iceberg documentation (#17938)
141f1d0475 is described below
commit 141f1d0475a77763458c5ac7bf38ee808e6dbf0b
Author: Talat UYARER <ta...@apache.org>
AuthorDate: Mon Sep 7 03:04:56 2026 -0700
Docs, Flink: Update Flink configuration and write options in Iceberg documentation (#17938)
---
docs/docs/flink-configuration.md | 27 ++++++++++++++++++++++++++-
docs/docs/flink-writes.md | 3 +--
docs/docs/flink.md | 3 ++-
3 files changed, 29 insertions(+), 4 deletions(-)
diff --git a/docs/docs/flink-configuration.md b/docs/docs/flink-configuration.md
index efaf59e02d..d0b4889177 100644
--- a/docs/docs/flink-configuration.md
+++ b/docs/docs/flink-configuration.md
@@ -37,7 +37,7 @@ The following properties can be set globally and are not limited to a specific c
| Property | Required | Values | Description |
| ---------------------------- |----------| -------------------------- |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| type | ✔️ | iceberg | Must be `iceberg`. |
-| catalog-type | | `hive`, `hadoop`, `rest`, `glue`, `jdbc` or `nessie` | The underlying Iceberg catalog implementation, `HiveCatalog`, `HadoopCatalog`, `RESTCatalog`, `GlueCatalog`, `JdbcCatalog`, `NessieCatalog` or left unset if using a custom catalog implementation via catalog-impl|
+| catalog-type | | `hive`, `hadoop` or `rest` | The underlying Iceberg catalog implementation, `HiveCatalog`, `HadoopCatalog` or `RESTCatalog`. Must be left unset when using a custom catalog implementation via `catalog-impl`, for example the AWS Glue, JDBC or Nessie catalogs.|
| catalog-impl | | | The fully-qualified class name of a custom catalog implementation. Must be set if `catalog-type` is unset. |
| property-version | | | Version number to describe the property version. This property can be used for backwards compatibility in case the property format changes. The current property version is `1`. |
| cache-enabled | | `true` or `false` | Whether to enable catalog cache, default value is `true`. |
@@ -159,9 +159,14 @@ INSERT INTO tableName /*+ OPTIONS('upsert-enabled'='true') */
| compression-level | Table write.(fileformat).compression-level | Overrides this table's compression level for Parquet and Avro tables for this write |
| compression-strategy | Table write.orc.compression-strategy | Overrides this table's compression strategy for ORC tables for this write |
| write-parallelism | Upstream operator parallelism | Overrides the writer parallelism |
+| branch | main | Branch to write to |
| uid-suffix | As per table property | Overrides the uid suffix used in the underlying IcebergSink for this table |
| shred-variants | Table write.parquet.shred-variants | Overrides this table's shred variants for this write |
| variant-inference-buffer-size | Table write.parquet.variant-inference-buffer-size | Overrides this table's variant inference buffer size for this write |
+| flink-maintenance.rewrite.enabled | false | Run data file compaction after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). |
+| flink-maintenance.expire-snapshots.enabled | false | Expire old snapshots after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). |
+| flink-maintenance.delete-orphan-files.enabled | false | Delete orphan files after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). |
+| flink-maintenance.convert-equality-deletes.enabled | false | Convert equality deletes to deletion vectors after successful commits. Only used by `IcebergSink`, see [post-commit table maintenance](flink-maintenance.md#icebergsink-with-post-commit-integration). |
#### Range distribution statistics type
@@ -201,3 +206,23 @@ they are.
This is only applicable to [`StatisticsType.Map`](../../javadoc/{{ icebergVersion }}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Map) for low-cardinality scenario. For [`StatisticsType.Sketch`](../../javadoc/{{ icebergVersion }}/org/apache/iceberg/flink/sink/shuffle/StatisticsType.html#Sketch) high-cardinality sort columns, they are usually not used as
partition columns. Otherwise, too many partitions and small files may be generated during
write. Sketch range partitioner simply splits high-cardinality keys into ordered ranges.
+
+### Execution options
+
+Iceberg also provides a set of `table.exec.iceberg.*` options that are read from the Flink
+configuration instead of per-job read or write options. In SQL they are set on the session:
+
+```sql
+SET 'table.exec.iceberg.infer-source-parallelism' = 'false';
+```
+
+When using the DataStream API, they can be set in the `Configuration` passed to the source or sink
+builder.
+
+| Flink configuration | Default | Description |
+|--------------------------------------------------|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| table.exec.iceberg.infer-source-parallelism | true | If true, the source parallelism for batch reads is inferred from the number of scan splits, capped by `table.exec.iceberg.infer-source-parallelism.max` and by the query limit if one is set. If false, the source parallelism is taken from the Flink configuration. Streaming reads never infer parallelism. |
+| table.exec.iceberg.infer-source-parallelism.max | 100 | Maximum inferred parallelism for the source operator. |
+| table.exec.iceberg.fetch-batch-record-count | 2048 | Target number of records per fetch batch in the Iceberg source reader. |
+| table.exec.iceberg.worker-pool-size | max(2, available cpu) | Size of the worker pool used to plan or scan manifests. Defaults to the shared Iceberg worker pool size, which is controlled by the `iceberg.worker.num-threads` system property. |
+| table.exec.iceberg.use-v2-sink | false | Use the SinkV2 based `IcebergSink` implementation, see [Sink V2 based implementation](flink-writes.md#sink-v2-based-implementation). |
diff --git a/docs/docs/flink-writes.md b/docs/docs/flink-writes.md
index 6ffedd5a77..e01736d69c 100644
--- a/docs/docs/flink-writes.md
+++ b/docs/docs/flink-writes.md
@@ -393,9 +393,8 @@ SET table.exec.iceberg.use-v2-sink = true;
To use SinkV2 based implementation, replace `FlinkSink` with `IcebergSink` in the provided snippets.
!!! warning
- There are some slight differences between these implementations:
+ There is a slight difference between these implementations:
- - The `RANGE` distribution mode is not yet available for the `IcebergSink`
- When using `IcebergSink` use `uidSuffix` instead of the `uidPrefix`
### Post-commit table maintenance
diff --git a/docs/docs/flink.md b/docs/docs/flink.md
index c42f5ea21f..1b62f15298 100644
--- a/docs/docs/flink.md
+++ b/docs/docs/flink.md
@@ -360,6 +360,7 @@ Flink types are converted to Iceberg types according to the following table:
| map | map | |
| multiset | map | |
| row | struct | |
+| variant | variant | Flink 2.x only |
| raw | | Not supported |
| interval | | Not supported |
| structured | | Not supported |
@@ -395,7 +396,7 @@ Iceberg types are converted to Flink types according to the following table:
| nanosecond timestamp | timestamp(9) | |
| nanosecond timestamp with timezone | timestamp_ltz(9) | |
| unknown | null | |
-| variant | | Not supported |
+| variant | variant | Flink 2.x only |
| geometry | | Not supported |
| geography | | Not supported |