You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by si...@apache.org on 2025/09/16 16:51:40 UTC
(iceberg) branch main updated: Docs: Update Spark Structured Streaming docs for Rate Limiting & Triggers (#14030)
This is an automated email from the ASF dual-hosted git repository.
singhpk234 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new b8c3e20722 Docs: Update Spark Structured Streaming docs for Rate Limiting & Triggers (#14030)
b8c3e20722 is described below
commit b8c3e20722728c8b16555e385d2c326d70a5d989
Author: Alex Prosak <14...@users.noreply.github.com>
AuthorDate: Tue Sep 16 09:51:35 2025 -0700
Docs: Update Spark Structured Streaming docs for Rate Limiting & Triggers (#14030)
---
docs/docs/spark-configuration.md | 29 ++++++++++++-----------------
docs/docs/spark-structured-streaming.md | 27 +++++++++++++++++++++++++++
2 files changed, 39 insertions(+), 17 deletions(-)
diff --git a/docs/docs/spark-configuration.md b/docs/docs/spark-configuration.md
index fdd90b943a..87e3386b58 100644
--- a/docs/docs/spark-configuration.md
+++ b/docs/docs/spark-configuration.md
@@ -211,23 +211,18 @@ spark.read
.table("catalog.db.table")
```
-| Spark option | Default | Description |
-| --------------- | --------------------- | ----------------------------------------------------------------------------------------- |
-| snapshot-id | (latest) | Snapshot ID of the table snapshot to read |
-| as-of-timestamp | (latest) | A timestamp in milliseconds; the snapshot used will be the snapshot current at this time. |
-| split-size | As per table property | Overrides this table's read.split.target-size and read.split.metadata-target-size |
-| lookback | As per table property | Overrides this table's read.split.planning-lookback |
-| file-open-cost | As per table property | Overrides this table's read.split.open-file-cost |
-| vectorization-enabled | As per table property | Overrides this table's read.parquet.vectorization.enabled |
-| batch-size | As per table property | Overrides this table's read.parquet.vectorization.batch-size |
-| stream-from-timestamp | (none) | A timestamp in milliseconds to stream from; if before the oldest known ancestor snapshot, the oldest will be used |
-| streaming-max-files-per-micro-batch | INT_MAX | Maximum number of files per microbatch |
-| streaming-max-rows-per-micro-batch | INT_MAX | Maximum number of rows per microbatch |
-
-!!! warning
- streaming-max-rows-per-micro-batch option sets a “soft max”, a batch will always include all the rows in the next unprocessed data file but additional files will not be included if doing so would exceed the soft max limit.
-
-
+| Spark option | Default | Description |
+| --------------- | --------------------- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| snapshot-id | (latest) | Snapshot ID of the table snapshot to read |
+| as-of-timestamp | (latest) | A timestamp in milliseconds; the snapshot used will be the snapshot current at this time. |
+| split-size | As per table property | Overrides this table's read.split.target-size and read.split.metadata-target-size |
+| lookback | As per table property | Overrides this table's read.split.planning-lookback |
+| file-open-cost | As per table property | Overrides this table's read.split.open-file-cost |
+| vectorization-enabled | As per table property | Overrides this table's read.parquet.vectorization.enabled |
+| batch-size | As per table property | Overrides this table's read.parquet.vectorization.batch-size |
+| stream-from-timestamp | (none) | A timestamp in milliseconds to stream from; if before the oldest known ancestor snapshot, the oldest will be used |
+| streaming-max-files-per-micro-batch | INT_MAX | Maximum number of files per microbatch |
+| streaming-max-rows-per-micro-batch | INT_MAX | "Soft maximum" number of rows per microbatch; always includes all rows in next unprocessed file, excludes additional files if their inclusion would exceed the soft max limit |
### Write options
diff --git a/docs/docs/spark-structured-streaming.md b/docs/docs/spark-structured-streaming.md
index 0ac753808d..07af8134c8 100644
--- a/docs/docs/spark-structured-streaming.md
+++ b/docs/docs/spark-structured-streaming.md
@@ -36,6 +36,33 @@ val df = spark.readStream
!!! warning
Iceberg only supports reading data from append snapshots. Overwrite snapshots cannot be processed and will cause an exception by default. Overwrites may be ignored by setting `streaming-skip-overwrite-snapshots=true`. Similarly, delete snapshots will cause an exception by default, and deletes may be ignored by setting `streaming-skip-delete-snapshots=true`.
+### Limit input rate
+To control the size of micro-batches in the DataFrame API, Iceberg supports two read options:
+
+* `streaming-max-files-per-micro-batch` Maximum number of files to be processed in every micro-batch.
+* `streaming-max-rows-per-micro-batch` A "soft max" on the number of rows to be processed in every micro-batch. A batch will always include all the rows in the next unprocessed data file but additional files will not be included if doing so would exceed the soft max limit.
+
+If both options are set, the micro-batch size will be limited by whichever option is reached first.
+
+```scala
+// Read a hard limit of 1 file per micro-batch
+val df = spark.readStream
+ .format("iceberg")
+ .option("streaming-max-files-per-micro-batch", "1")
+ .load("database.table_name")
+```
+
+```scala
+// Read files until the number of included rows >= 1000 per micro-batch
+val df = spark.readStream
+ .format("iceberg")
+ .option("streaming-max-rows-per-micro-batch", "1000")
+ .load("database.table_name")
+```
+
+!!! info
+ Note: In addition to limiting micro-batch sizes on queries that use the default trigger (i.e. `Trigger.ProcessingTime`), rate limiting options can be applied to queries that use `Trigger.AvailableNow` to split one-time processing of all available source data into multiple micro-batches for better query scalability. Rate limiting options will be ignored when using the deprecated `Trigger.Once` trigger.
+
## Streaming Writes
To write values from streaming query to Iceberg table, use `DataStreamWriter`: