You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluss.apache.org by yu...@apache.org on 2026/04/24 03:40:51 UTC
(fluss) branch main updated: [spark][doc] Add Spark batch union read (#3142)
This is an automated email from the ASF dual-hosted git repository.
luoyuxia pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fluss.git
The following commit(s) were added to refs/heads/main by this push:
new b6d2f332a [spark][doc] Add Spark batch union read (#3142)
b6d2f332a is described below
commit b6d2f332a686fbfb42b815605794591b588318be
Author: Yang Zhang <ya...@alibaba-inc.com>
AuthorDate: Fri Apr 24 11:40:48 2026 +0800
[spark][doc] Add Spark batch union read (#3142)
---
website/docs/engine-spark/getting-started.md | 2 +-
website/docs/engine-spark/reads.md | 63 +++++++++++++++++++++++++---
website/docs/streaming-lakehouse/overview.md | 2 +-
3 files changed, 59 insertions(+), 8 deletions(-)
diff --git a/website/docs/engine-spark/getting-started.md b/website/docs/engine-spark/getting-started.md
index e93fe0c86..7d1b0d5cf 100644
--- a/website/docs/engine-spark/getting-started.md
+++ b/website/docs/engine-spark/getting-started.md
@@ -28,7 +28,7 @@ Fluss supports Apache Spark's SQL API and Spark Structured Streaming.
| [SQL Show Partitions](ddl.md#show-partitions) | ✔️ | |
| [SQL Add Partition](ddl.md#add-partition) | ✔️ | |
| [SQL Drop Partition](ddl.md#drop-partition) | ✔️ | |
-| [SQL Select (Batch)](reads.md) | ✔️ | Log table and primary-key table |
+| [SQL Select (Batch)](reads.md) | ✔️ | Log table and primary-key table; both support union read when lake-enabled |
| [SQL Insert Into](writes.md) | ✔️ | Log table and primary-key table |
| [Structured Streaming Read](structured-streaming.md#streaming-read) | ✔️ | Log table and primary-key table |
| [Structured Streaming Write](structured-streaming.md#streaming-write) | ✔️ | Log table and primary-key table |
diff --git a/website/docs/engine-spark/reads.md b/website/docs/engine-spark/reads.md
index 26cc1c5fd..c9d745e45 100644
--- a/website/docs/engine-spark/reads.md
+++ b/website/docs/engine-spark/reads.md
@@ -12,12 +12,6 @@ Fluss supports batch read with [Apache Spark](https://spark.apache.org/)'s SQL A
For streaming read, see the [Structured Streaming Read](structured-streaming.md#streaming-read) section.
:::
-## Limitations
-
-:::caution
-- For tables with datalake enabled (`table.datalake.enabled = true`), batch read can only read data stored in the Fluss cluster and cannot read data that has been tiered to the underlying lake storage (e.g., Paimon, Iceberg). Reading the full dataset including lake data will be supported in a future release.
-:::
-
## Batch Read
### Log Table
@@ -203,6 +197,63 @@ WHERE dt IN ('2026-01-01', '2026-01-02')
ORDER BY order_id;
```
+### Lake-Enabled Tables (Union Read)
+
+When a table has the configuration `table.datalake.enabled = 'true'`, its data exists in two layers:
+
+- **Fresh data** is retained in Fluss (real-time layer, sub-second freshness)
+- **Historical data** is tiered to the lake storage (e.g., Paimon, Iceberg)
+
+Fluss Spark connector supports **union read** that combines both layers to provide a complete, up-to-date view of the data. This allows Fluss to store only a small portion of the dataset in the cluster (reducing costs), while the lake serves as the source of complete historical data.
+
+#### Union Read
+
+To read the full dataset, simply query the table directly. The Spark connector automatically unions data from Fluss and the lake storage:
+
+```sql title="Spark SQL"
+-- Query will union data from Fluss and lake
+SELECT SUM(total_price) AS total_revenue FROM fluss_order_with_lake;
+```
+
+The union read works for both **log tables** and **primary key tables**:
+
+- **Log tables**: Combines Fluss log data with lake historical data
+- **Primary key tables**: Combines lake snapshot data with recent KV log changes using sort-merge to provide the most up-to-date view
+
+#### Example
+
+1. Create a lake-enabled table:
+
+```sql title="Spark SQL"
+CREATE TABLE fluss_order_with_lake (
+ order_key BIGINT,
+ cust_key INT NOT NULL,
+ total_price DECIMAL(15, 2),
+ order_date DATE,
+ order_priority STRING,
+ clerk STRING
+) TBLPROPERTIES (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+2. Insert data (the datalake tiering service will continuously tier data to the lake):
+
+```sql title="Spark SQL"
+INSERT INTO fluss_order_with_lake VALUES
+ (1001, 101, 150.50, DATE '2026-01-01', 'HIGH', 'clerk_A'),
+ (1002, 102, 250.75, DATE '2026-01-01', 'MEDIUM', 'clerk_B'),
+ (1003, 103, 350.00, DATE '2026-01-02', 'LOW', 'clerk_C');
+```
+
+3. Query with union read:
+
+```sql title="Spark SQL"
+-- Returns complete view combining Fluss and lake data
+SELECT SUM(total_price) AS total_revenue FROM fluss_order_with_lake;
+```
+
## All Data Types
Fluss Spark connector supports reading all Fluss data types including nested types:
diff --git a/website/docs/streaming-lakehouse/overview.md b/website/docs/streaming-lakehouse/overview.md
index 626c6ae0e..e501b3bc3 100644
--- a/website/docs/streaming-lakehouse/overview.md
+++ b/website/docs/streaming-lakehouse/overview.md
@@ -38,7 +38,7 @@ The core idea of Streaming Lakehouse is shared data and shared metadata between
Some powerful features it provides are:
- **Unified Metadata**: Fluss provides unified table metadata for both data in Stream and Lakehouse. Users only need to manage one table and can access real-time streaming data, historical data, or both combined.
-- **Union Reads**: Compute engines that perform queries on the table will read the union of the real-time streaming data and Lakehouse data. Currently, only Flink supports union reads, but more engines are on the roadmap.
+- **Union Reads**: Compute engines that perform queries on the table will read the union of the real-time streaming data and Lakehouse data. Currently, Flink and Spark support union reads, with more engines on the roadmap.
- **Real-Time Lakehouse**: The union reads help Lakehouse evolving from near-real-time analytics to truly real-time analytics. This empowers businesses to gain more valuable insights from real-time data.
- **Analytical Streams**: The union reads help data streams to have the powerful analytics capabilities. This reduces complexity when developing streaming applications, simplifies debugging, and allows for immediate access to live data insights.
- **Connect to Lakehouse Ecosystem**: Fluss keeps the table metadata in sync with data lake catalogs while compacting data into Lakehouse. As a result, external engines like Spark, StarRocks, Flink, and Trino can read the data directly. They simply connect to the data lake catalog.