You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluss.apache.org by ab...@apache.org on 2026/07/24 16:13:45 UTC
(fluss) 02/02: [docs] hive and glue fixes and adjustments
This is an automated email from the ASF dual-hosted git repository.
fresh-borzoni pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fluss.git
commit d688bbd7216c60ff2a58e86a55c8bf9486211b51
Author: Anton Borisov <an...@fresha.com>
AuthorDate: Fri Jul 24 16:21:51 2026 +0100
[docs] hive and glue fixes and adjustments
---
.../streaming-lakehouse/datalake-catalogs/glue.md | 83 +++++++++++-----------
.../streaming-lakehouse/datalake-catalogs/hive.md | 42 ++++++-----
.../datalake-formats/iceberg.md | 6 +-
website/src/css/custom.css | 13 ++++
4 files changed, 82 insertions(+), 62 deletions(-)
diff --git a/website/docs/streaming-lakehouse/datalake-catalogs/glue.md b/website/docs/streaming-lakehouse/datalake-catalogs/glue.md
index 24f2fdbd2..57f247efc 100644
--- a/website/docs/streaming-lakehouse/datalake-catalogs/glue.md
+++ b/website/docs/streaming-lakehouse/datalake-catalogs/glue.md
@@ -1,6 +1,6 @@
---
title: AWS Glue
-sidebar_position: 2
+sidebar_position: 3
---
# AWS Glue
@@ -15,9 +15,9 @@ This guide explains how to configure Fluss to use AWS Glue as its Iceberg catalo
When Fluss is configured with AWS Glue as its Iceberg catalog:
-1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python) or Flink SQL. Fluss stores this data in its log tables.
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python) or Flink SQL. Fluss stores this data in its tables (log tables or primary key tables).
2. **Tiering to Iceberg**: A separate Flink job (the tiering service) periodically reads accumulated data from Fluss, converts it to Parquet format, writes the files to S3, and commits an Iceberg snapshot to the Glue Data Catalog.
-3. **Query via Athena/Spark/Trino**: Any Iceberg-compatible engine can discover and query the tiered tables through AWS Glue no additional configuration needed.
+3. **Query via Athena/Spark/Trino**: Any Iceberg-compatible engine can discover and query the tiered tables through AWS Glue — no additional configuration needed.
```mermaid
flowchart TD
@@ -51,15 +51,18 @@ flowchart TD
D3 -->|commit snapshot| C1
C1 -.->|discover tables| C3
C2 -.->|read data| C3
+ B2 -->|remote log & KV snapshots| C2
```
-> **Note**: Flink is used here for the tiering service (Fluss Iceberg/S3/Glue) and optionally for data ingestion via SQL. Applications can also write directly to Fluss using the client library (Java, Python). The tiering service is what bridges Fluss's log tables to the Glue Data Catalog.
+> **Note**: Flink is used here for the tiering service (Fluss → Iceberg/S3/Glue) and optionally for data ingestion via SQL. Applications can also write directly to Fluss using the client library (Java, Python). The tiering service is what bridges Fluss tables to the Glue Data Catalog.
## Prerequisites
### Java Version
-Fluss 0.9.x requires **Java 17 or later** for the Tablet Server. Java 11 will fail with `NoSuchMethodError: MappedByteBuffer.duplicate()`.
+Fluss requires **Java 11 or later**, with Java 17 recommended (see [Deploying Fluss](../../install-deploy/deploying-local-cluster.md)).
+
+> **NOTE**: The 0.9.1 release binaries do not run on Java 11 — the Tablet Server fails with `NoSuchMethodError: MappedByteBuffer.duplicate()` because the release was compiled on a newer JDK. Use Java 17+ with the 0.9.1 binaries, or build Fluss from source on JDK 11.
### AWS IAM Permissions
@@ -119,32 +122,32 @@ Place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/` directory:
- **Iceberg AWS Bundle**: [iceberg-aws-bundle-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.1/iceberg-aws-bundle-1.10.1.jar)
- **Iceberg AWS**: [iceberg-aws-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws/1.10.1/iceberg-aws-1.10.1.jar)
-- **Failsafe**: [failsafe-3.3.2.jar](https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar) required by `iceberg-aws` for Glue API retry logic
+- **Failsafe**: [failsafe-3.3.2.jar](https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar) — required by `iceberg-aws` for Glue API retry logic
Both Iceberg JARs are required. The bundle provides AWS SDK v2 dependencies, while `iceberg-aws` provides the `GlueCatalog` class that Iceberg loads via reflection. The plugin classloader cannot find `GlueCatalog` from the bundle alone. The `failsafe` library is a transitive dependency of `iceberg-aws` that is not bundled.
> **NOTE**: You need **all three** JARs. Using only the bundle will result in `ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog`. Missing `failsafe` will cause `NoClassDefFoundError: dev/failsafe/FailsafeException`.
-Additionally, you need the Fluss S3 filesystem plugin for remote storage access. Place [fluss-fs-s3-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-fs-s3/$FLUSS_VERSION$/fluss-fs-s3-$FLUSS_VERSION$.jar) in `${FLUSS_HOME}/plugins/s3/`. See [S3 Dependencies](/docs/maintenance/filesystems/s3#dependencies) for details.
+Additionally, you need the Fluss S3 filesystem plugin for remote storage access. Place [fluss-fs-s3-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-fs-s3/$FLUSS_VERSION$/fluss-fs-s3-$FLUSS_VERSION$.jar) in `${FLUSS_HOME}/plugins/s3/`. See [S3 Dependencies](../../maintenance/tiered-storage/filesystems/s3.md#dependencies) for details.
-> **TIP**: The Fluss binary distribution already includes `fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need to download it separately only add the three JARs above (iceberg-aws-bundle, iceberg-aws, failsafe).
+> **TIP**: The Fluss binary distribution already includes `fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need to download it separately — only add the three JARs above (iceberg-aws-bundle, iceberg-aws, failsafe).
#### For the Flink Tiering Service
Place the following JARs in `${FLINK_HOME}/lib`:
1. **Iceberg AWS Bundle**: `iceberg-aws-bundle-1.10.1.jar` (same as above)
-2. **Iceberg AWS**: `iceberg-aws-1.10.1.jar` (same as above provides `GlueCatalog` class)
-3. **Failsafe**: `failsafe-3.3.2.jar` (same as above Glue retry logic)
+2. **Iceberg AWS**: `iceberg-aws-1.10.1.jar` (same as above — provides `GlueCatalog` class)
+3. **Failsafe**: `failsafe-3.3.2.jar` (same as above — Glue retry logic)
4. **Fluss Flink Connector**: [fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar) (pick the version matching your Flink runtime)
5. **Fluss Lake Iceberg**: [fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
-6. **Fluss Flink Tiering**: [fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar) the tiering job JAR itself
+6. **Fluss Flink Tiering**: [fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar) — the tiering job JAR itself
7. **Fluss S3 Filesystem**: [fluss-fs-s3-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-fs-s3/$FLUSS_VERSION$/fluss-fs-s3-$FLUSS_VERSION$.jar) (if S3 is used as Fluss remote storage)
-8. **Hadoop Client**: [hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar) and [hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar) required by the Iceberg Parquet writer
+8. **Hadoop Client**: [hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar) and [hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar) — required by the Iceberg Parquet writer
> **NOTE**: Despite the Glue catalog itself not requiring Hadoop, the **tiering service** needs Hadoop classes (`org.apache.hadoop.conf.Configuration`) for writing Parquet files via Iceberg. Use the `hadoop-client-api` and `hadoop-client-runtime` JARs (not the full Hadoop distribution or `flink-shaded-hadoop-2-uber`) to avoid classpath conflicts with Flink's bundled Avro version. Using the uber JAR causes `NoSuchMethodError: LogicalTypes.timestampNanos()`.
-> **WARNING**: Do **not** add S3 credentials to Flink's configuration file (`config.yaml` or `flink-conf.yaml`) by appending key-value pairs this can break Flink's memory configuration parser. The credential provider chain configured in Fluss's `server.yaml` handles S3 access for the tiering service automatically.
+> **WARNING**: Do **not** add S3 credentials to Flink's configuration file (`config.yaml` or `flink-conf.yaml`) by appending key-value pairs — this can break Flink's memory configuration parser. The credential provider chain configured in Fluss's `server.yaml` handles S3 access for the tiering service automatically.
## Configure Fluss with AWS Glue
@@ -169,8 +172,8 @@ datalake.iceberg.client.region: us-east-1
# Use S3FileIO explicitly (Glue catalog defaults to this when warehouse is s3://)
datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
-# Optional: restrict catalog to a specific Glue database prefix
-datalake.iceberg.glue.catalog-id: <aws-account-id>
+# Optional: use a Glue Data Catalog in another AWS account (defaults to the caller's account)
+datalake.iceberg.glue.id: <aws-account-id>
```
#### Authentication
@@ -198,7 +201,7 @@ Example for ECS Fargate:
fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider
```
-The credential provider chain handles automatic credential refresh no static keys or manual token rotation needed.
+The credential provider chain handles automatic credential refresh — no static keys or manual token rotation needed.
> **NOTE**: For non-standard AWS regions (GovCloud, China), you must also set the S3 endpoint explicitly. Hadoop S3A defaults to `s3.amazonaws.com` which does not work for these partitions:
> ```yaml
@@ -206,7 +209,7 @@ The credential provider chain handles automatic credential refresh no static ke
> fs.s3a.endpoint: s3.cn-north-1.amazonaws.com.cn # China
> ```
-> **NOTE**: You may see a non-fatal warning: `"Session credentials from the configured AWS credentials provider are not supported for Fluss S3 client-token generation."` This is expected the delegation token subsystem cannot generate tokens from session credentials, but the actual S3 file operations work correctly via the provider chain.
+> **NOTE**: You may see a non-fatal warning: `"Session credentials from the configured AWS credentials provider are not supported for Fluss S3 client-token generation."` This is expected — the delegation token subsystem cannot generate tokens from session credentials, but the actual S3 file operations work correctly via the provider chain.
**Static Credentials (Testing Only)**:
@@ -237,11 +240,11 @@ ${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
This section shows the complete setup from a blank Linux machine to data queryable in Athena. Adapt paths and versions for your environment.
```bash
-# 1. Install prerequisites
+# — 1. Install prerequisites —
# Java 17+ required (Amazon Corretto, OpenJDK, etc.)
java -version # Must show 17+
-# 2. Install ZooKeeper
+# — 2. Install ZooKeeper —
wget -q https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
tar -xzf apache-zookeeper-3.8.4-bin.tar.gz -C /opt/zookeeper --strip-components=1
cat > /opt/zookeeper/conf/zoo.cfg <<EOF
@@ -250,11 +253,11 @@ dataDir=/var/lib/zookeeper
clientPort=2181
EOF
-# 3. Install Fluss
+# — 3. Install Fluss —
wget -q https://dlcdn.apache.org/incubator/fluss/fluss-0.9.1-incubating/fluss-0.9.1-incubating-bin.tgz
tar -xzf fluss-0.9.1-incubating-bin.tgz -C /opt/fluss --strip-components=1
-# 4. Add Glue JARs to Fluss plugins/iceberg/
+# — 4. Add Glue JARs to Fluss plugins/iceberg/ —
wget -O /opt/fluss/plugins/iceberg/iceberg-aws-bundle-1.10.1.jar \
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.1/iceberg-aws-bundle-1.10.1.jar
wget -O /opt/fluss/plugins/iceberg/iceberg-aws-1.10.1.jar \
@@ -262,7 +265,7 @@ wget -O /opt/fluss/plugins/iceberg/iceberg-aws-1.10.1.jar \
wget -O /opt/fluss/plugins/iceberg/failsafe-3.3.2.jar \
https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar
-# 5. Write server.yaml
+# — 5. Write server.yaml —
cat > /opt/fluss/conf/server.yaml <<EOF
zookeeper.address: localhost:2181
coordinator.host: localhost
@@ -279,7 +282,7 @@ datalake.iceberg.warehouse: s3://YOUR-BUCKET/iceberg-warehouse
datalake.iceberg.client.region: YOUR-REGION
datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
-# S3 credentials use the provider chain for your environment
+# S3 credentials — use the provider chain for your environment
# ECS Fargate:
fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider
# EKS (IRSA): use WebIdentityTokenCredentialsProvider instead of ContainerCredentialsProvider
@@ -289,7 +292,7 @@ fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider
# fs.s3a.endpoint: s3.YOUR-REGION.amazonaws.com
EOF
-# 6. Start services
+# — 6. Start services —
/opt/zookeeper/bin/zkServer.sh start
sleep 3
/opt/fluss/bin/coordinator-server.sh start
@@ -297,7 +300,7 @@ sleep 5
/opt/fluss/bin/tablet-server.sh start
sleep 5
-# 7. Install Flink + tiering JARs
+# — 7. Install Flink + tiering JARs —
wget -q https://archive.apache.org/dist/flink/flink-1.20.1/flink-1.20.1-bin-scala_2.12.tgz
tar -xzf flink-1.20.1-bin-scala_2.12.tgz -C /opt/flink --strip-components=1
@@ -320,7 +323,7 @@ wget -O /opt/flink/lib/hadoop-client-runtime-3.3.6.jar \
/opt/flink/bin/start-cluster.sh
sleep 5
-# 8. Create table + insert data (single Flink SQL session)
+# — 8. Create table + insert data (single Flink SQL session) —
cat > /tmp/setup.sql <<EOF
CREATE CATALOG fluss_catalog WITH ('type' = 'fluss', 'bootstrap.servers' = 'localhost:9123');
USE CATALOG fluss_catalog;
@@ -332,7 +335,7 @@ INSERT INTO test_table VALUES (1, 'hello'), (2, 'world');
EOF
/opt/flink/bin/sql-client.sh -f /tmp/setup.sql
-# 9. Start tiering service
+# — 9. Start tiering service —
/opt/flink/bin/flink run -d /opt/flink/lib/fluss-flink-tiering-0.9.1-incubating.jar \
--fluss.bootstrap.servers localhost:9123 \
--datalake.format iceberg \
@@ -341,7 +344,7 @@ EOF
--datalake.iceberg.client.region YOUR-REGION \
--datalake.iceberg.io-impl org.apache.iceberg.aws.s3.S3FileIO
-# 10. Query in Athena
+# — 10. Query in Athena —
# Wait ~30s for tiering to flush, then:
# SELECT * FROM my_database.test_table;
```
@@ -376,7 +379,7 @@ CREATE TABLE customer_orders (
Fluss will register the database in ZooKeeper and create a corresponding Iceberg table in the Glue Data Catalog (using the same database name). Once data is ingested and the tiering service commits, Parquet files appear in the S3 warehouse path.
-> **NOTE**: The database must be created in the Fluss catalog first Fluss maintains its own database registry in ZooKeeper. The database name in Fluss maps 1:1 to the Glue database name during tiering.
+> **NOTE**: The database must be created in the Fluss catalog first — Fluss maintains its own database registry in ZooKeeper. The database name in Fluss maps 1:1 to the Glue database name during tiering.
### Query Data with Athena
@@ -389,15 +392,15 @@ SELECT * FROM my_database.customer_orders LIMIT 10;
### Query Data with Flink (Union Read)
```sql title="Flink SQL"
-SET 'execution.runtime-mode' = 'batch';
+SET 'execution.runtime-mode' = 'streaming';
--- Union read: combines fresh data in Fluss with historical data in Iceberg
-SELECT COUNT(*) FROM customer_orders;
+-- Union read: starts from the Iceberg snapshot in Glue, then continues with the Fluss changelog
+SELECT * FROM customer_orders;
```
-> **NOTE**: Union reads for **primary key tables** with the Iceberg lake format are not yet supported. If your table has a primary key, query tiered data directly through Athena/Spark (which reads from the Iceberg snapshot in Glue) or read from the Fluss log only. Union reads work for log tables (append-only, no primary key).
+> **NOTE**: For **primary key tables**, union reads work in **streaming** mode only — **batch** mode is not yet supported with the Iceberg lake format and fails with `UnsupportedOperationException`. In batch you can only query the **tiered data** — which excludes recent changes not yet tiered — either through Athena/Spark against the Iceberg table in Glue, or via `SELECT ... FROM customer_orders$lake` in Flink. Log tables (append-only, no primary key) support union reads in both modes.
-For details on union reads and streaming reads, see [Iceberg - Read Tables](../datalake-formats/iceberg.md#read-tables).
+For details on union reads and streaming reads, see [Union Read](../union-read.md) and [Iceberg - Read Tables](../datalake-formats/iceberg.md#read-tables).
## Troubleshooting
@@ -407,8 +410,8 @@ For details on union reads and streaming reads, see [Iceberg - Read Tables](../d
| `NoClassDefFoundError: dev/failsafe/FailsafeException` | Missing `failsafe` JAR | Add `failsafe-3.3.2.jar` to `plugins/iceberg/` and `${FLINK_HOME}/lib` |
| `NoClassDefFoundError: software/amazon/awssdk/...` | Missing AWS SDK | Ensure `iceberg-aws-bundle-1.10.1.jar` is in `plugins/iceberg/` and `${FLINK_HOME}/lib` |
| `NoClassDefFoundError: org/apache/hadoop/conf/Configuration` | Missing Hadoop JARs in tiering | Add `hadoop-client-api-3.3.6.jar` and `hadoop-client-runtime-3.3.6.jar` to `${FLINK_HOME}/lib` |
-| `NoSuchMethodError: LogicalTypes.timestampNanos()` | Avro version conflict | Do NOT use `flink-shaded-hadoop-2-uber` use `hadoop-client-api` + `hadoop-client-runtime` instead |
-| `NoSuchMethodError: MappedByteBuffer.duplicate()` | Java version too old | Fluss 0.9.x requires **Java 17+**. Upgrade from Java 11. |
+| `NoSuchMethodError: LogicalTypes.timestampNanos()` | Avro version conflict | Do NOT use `flink-shaded-hadoop-2-uber` — use `hadoop-client-api` + `hadoop-client-runtime` instead |
+| `NoSuchMethodError: MappedByteBuffer.duplicate()` | 0.9.1 release binaries running on Java 11 | Use Java 17+ with the release binaries, or build Fluss from source on JDK 11 |
| `NoAwsCredentialsException` / `No AWS Credentials` | S3 plugin missing credential provider config | Set `fs.s3a.aws.credentials.provider` in `server.yaml` with the appropriate provider for your environment (see Authentication section above). |
| `SecurityTokenException: Region is not set` | S3 token manager missing region | Add `datalake.iceberg.client.region` and set `AWS_REGION` environment variable |
| `AccessDeniedException` from Glue API | Insufficient IAM or Lake Formation permissions | Check IAM policy includes `glue:CreateTable`. If Lake Formation is enabled, grant table-level permissions to the role. |
@@ -418,7 +421,7 @@ For details on union reads and streaming reads, see [Iceberg - Read Tables](../d
## Further Reading
-- [Iceberg Integration](../datalake-formats/iceberg.md) Table mapping, data types, supported catalog types, and limitations
-- [Streaming Lakehouse Overview](../overview.md) — General tiered storage concepts
-- [S3 Filesystem](/docs/maintenance/filesystems/s3) Configuring S3 as Fluss remote storage
-- [Iceberg AWS Docs](https://iceberg.apache.org/docs/1.10.1/aws/) Full reference for Glue and S3 properties
+- [Iceberg Integration](../datalake-formats/iceberg.md) — Table mapping, data types, supported catalog types, and limitations
+- [Streaming Lakehouse Overview](../overview.mdx) — General tiered storage concepts
+- [S3 Filesystem](../../maintenance/tiered-storage/filesystems/s3.md) — Configuring S3 as Fluss remote storage
+- [Iceberg AWS Docs](https://iceberg.apache.org/docs/1.10.1/aws/) — Full reference for Glue and S3 properties
diff --git a/website/docs/streaming-lakehouse/datalake-catalogs/hive.md b/website/docs/streaming-lakehouse/datalake-catalogs/hive.md
index e1c8de0ad..34d9695fe 100644
--- a/website/docs/streaming-lakehouse/datalake-catalogs/hive.md
+++ b/website/docs/streaming-lakehouse/datalake-catalogs/hive.md
@@ -1,6 +1,6 @@
---
title: Hive Metastore
-sidebar_position: 3
+sidebar_position: 4
---
# Hive Metastore
@@ -15,7 +15,7 @@ This guide explains how to configure Fluss to use Hive Metastore as its Iceberg
When Fluss is configured with Hive Metastore as its Iceberg catalog:
-1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python) or Flink SQL. Fluss stores this data in its log tables.
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python) or Flink SQL. Fluss stores this data in its tables (log tables or primary key tables).
2. **Tiering to Iceberg**: A separate Flink job (the tiering service) periodically reads accumulated data from Fluss, converts it to Parquet format, writes the files to HDFS (or S3/OSS), and commits an Iceberg snapshot to the Hive Metastore.
3. **Query via Spark/Trino/Flink**: Any Iceberg-compatible engine configured with Hive catalog can discover and query the tiered tables through HMS.
@@ -51,15 +51,18 @@ flowchart TD
D3 -->|commit snapshot| C1
C1 -.->|discover tables| C3
C2 -.->|read data| C3
+ B2 -->|remote log & KV snapshots| C2
```
-> **Note**: The tiering service is a Flink job that bridges Fluss's log tables to Iceberg tables in Hive Metastore. Flink is also commonly used for data ingestion (via SQL), but applications can write directly to Fluss using the client library.
+> **Note**: The tiering service is a Flink job that bridges Fluss tables to Iceberg tables in Hive Metastore. Flink is also commonly used for data ingestion (via SQL), but applications can write directly to Fluss using the client library.
## Prerequisites
### Java Version
-Fluss 0.9.x requires **Java 17 or later** for the Tablet Server.
+Fluss requires **Java 11 or later**, with Java 17 recommended (see [Deploying Fluss](../../install-deploy/deploying-local-cluster.md)).
+
+> **NOTE**: The 0.9.1 release binaries do not run on Java 11 — the Tablet Server fails with `NoSuchMethodError: MappedByteBuffer.duplicate()` because the release was compiled on a newer JDK. Use Java 17+ with the 0.9.1 binaries, or build Fluss from source on JDK 11.
### Running Hive Metastore
@@ -88,7 +91,7 @@ Download and place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/` di
| 7 | [jackson-databind-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar) | 1.5 MB | JSON data binding |
| 8 | [jackson-annotations-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar) | 75 KB | JSON annotations |
-> **NOTE**: The Fluss binary distribution already includes `fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need to download it separately only add the 8 JARs above.
+> **NOTE**: The Fluss binary distribution already includes `fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need to download it separately — only add the 8 JARs above.
> **TIP**: `hive-exec-2.3.10.jar` is a shaded uber JAR that bundles the Hive metastore client, `HiveConf`, Apache Thrift, libfb303, and Guava. Using the full `hive-exec` (not the `-core` variant) avoids needing to track down each transitive dependency individually. Despite its size, it is the simplest way to satisfy all Hive client requirements.
@@ -99,7 +102,7 @@ Place the following JARs in `${FLINK_HOME}/lib`:
1. **All 8 JARs listed above** (iceberg-hive-metastore, hive-exec, hadoop-client-api, hadoop-client-runtime, commons-logging, jackson-core, jackson-databind, jackson-annotations)
2. **Fluss Flink Connector**: [fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar) (pick the version matching your Flink runtime)
3. **Fluss Lake Iceberg**: [fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
-4. **Fluss Flink Tiering**: [fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar) the tiering job JAR itself
+4. **Fluss Flink Tiering**: [fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar) — the tiering job JAR itself
### Hadoop Classpath Configuration (HDFS Only)
@@ -151,7 +154,7 @@ ${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
## Quick Start (Docker Compose)
-This section provides a complete docker-compose setup that runs the entire Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink, creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back. No cloud account or Hadoop cluster needed everything runs locally.
+This section provides a complete docker-compose setup that runs the entire Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink, creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back. No cloud account or Hadoop cluster needed — everything runs locally.
**Prerequisites**: Docker and Docker Compose installed.
@@ -211,7 +214,7 @@ services:
- fluss-plugins:/plugins
coordinator:
- image: apache/fluss:$FLUSS_VERSION$
+ image: apache/fluss:$FLUSS_DOCKER_VERSION$
command: coordinatorServer
depends_on:
zookeeper: { condition: service_healthy }
@@ -239,7 +242,7 @@ services:
start_period: 30s
tablet-server:
- image: apache/fluss:$FLUSS_VERSION$
+ image: apache/fluss:$FLUSS_DOCKER_VERSION$
command: tabletServer
depends_on:
coordinator: { condition: service_healthy }
@@ -359,7 +362,7 @@ Run it:
docker compose up
```
-Watch the `flink` container logs. After about 3ΓÇô4 minutes (JAR downloads + tiering flush), you should see:
+Watch the `flink` container logs. After about 3–4 minutes (JAR downloads + tiering flush), you should see:
```
+----+----------+------------+----------+------------+
@@ -408,7 +411,7 @@ CREATE TABLE daily_events (
Fluss will create a corresponding Iceberg table in the Hive Metastore under the database matching your Fluss namespace. Once data is ingested and the tiering service commits a snapshot, Parquet files appear in the warehouse path and the table becomes queryable via HMS.
-> **NOTE**: The database must be created in the Fluss catalog first Fluss maintains its own database registry in ZooKeeper. The database name in Fluss maps 1:1 to the HMS database name during tiering.
+> **NOTE**: The database must be created in the Fluss catalog first — Fluss maintains its own database registry in ZooKeeper. The database name in Fluss maps 1:1 to the HMS database name during tiering.
### Query Data with Spark
@@ -422,14 +425,15 @@ SELECT * FROM hive_catalog.my_database.daily_events;
### Query Data with Flink (Union Read)
```sql title="Flink SQL"
--- Union read: combines fresh data in Fluss with historical data in Iceberg
SET 'execution.runtime-mode' = 'streaming';
+
+-- Union read: starts from the Iceberg snapshot in Hive Metastore, then continues with the Fluss changelog
SELECT * FROM daily_events;
```
-> **NOTE**: Union reads for **primary key tables** with the Iceberg lake format are not yet supported. If your table has a primary key, query tiered data directly through Spark/Trino (which reads from the Iceberg snapshot in Hive Metastore) or read from the Fluss log only. Union reads work for log tables (append-only, no primary key).
+> **NOTE**: For **primary key tables**, union reads work in **streaming** mode only — **batch** mode is not yet supported with the Iceberg lake format and fails with `UnsupportedOperationException`. In batch you can only query the **tiered data** — which excludes recent changes not yet tiered — either through Spark/Trino against the Iceberg table in Hive Metastore, or via `SELECT ... FROM daily_events$lake` in Flink. Log tables (append-only, no primary key) support union reads in both modes.
-For details on union reads and streaming reads, see [Iceberg - Read Tables](../datalake-formats/iceberg.md#read-tables).
+For details on union reads and streaming reads, see [Union Read](../union-read.md) and [Iceberg - Read Tables](../datalake-formats/iceberg.md#read-tables).
## Troubleshooting
@@ -442,11 +446,11 @@ For details on union reads and streaming reads, see [Iceberg - Read Tables](../d
| `ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException` | Missing Jackson | Add `jackson-core`, `jackson-databind`, `jackson-annotations` JARs to `plugins/iceberg/` |
| `TApplicationException: Invalid method name: 'get_table'` | HMS 4.x incompatibility | Use HMS **2.x or 3.x** (not 4.x). Iceberg's Hive client targets Hive 2.3.x which is incompatible with the HMS 4.x thrift protocol |
| `NoClassDefFoundError: com/google/common/collect/ImmutableMap` | Missing Guava | Ensure you're using the full `hive-exec-2.3.10.jar` (it bundles Guava internally) |
-| `NoSuchMethodError: MappedByteBuffer.duplicate()` | Java version too old | Fluss 0.9.x requires **Java 17+**. Upgrade from Java 11. |
+| `NoSuchMethodError: MappedByteBuffer.duplicate()` | 0.9.1 release binaries running on Java 11 | Use Java 17+ with the release binaries, or build Fluss from source on JDK 11 |
## Further Reading
-- [Iceberg Integration](../datalake-formats/iceberg.md) Table mapping, data types, and configurations.
-- [Lakehouse Storage](maintenance/tiered-storage/lakehouse-storage.md) General tiered storage overview.
-- [AWS Glue Catalog](glue.md) Alternative Iceberg catalog for AWS environments.
-- [Iceberg Hive Catalog Docs](https://iceberg.apache.org/docs/latest/hive/) Official Iceberg Hive documentation.
+- [Iceberg Integration](../datalake-formats/iceberg.md) — Table mapping, data types, and configurations.
+- [Lakehouse Storage](maintenance/tiered-storage/lakehouse-storage.md) — General tiered storage overview.
+- [AWS Glue Catalog](glue.md) — Alternative Iceberg catalog for AWS environments.
+- [Iceberg Hive Catalog Docs](https://iceberg.apache.org/docs/latest/hive/) — Official Iceberg Hive documentation.
diff --git a/website/docs/streaming-lakehouse/datalake-formats/iceberg.md b/website/docs/streaming-lakehouse/datalake-formats/iceberg.md
index 48f1eb93d..76198fb71 100644
--- a/website/docs/streaming-lakehouse/datalake-formats/iceberg.md
+++ b/website/docs/streaming-lakehouse/datalake-formats/iceberg.md
@@ -58,10 +58,10 @@ This approach enables passing custom configurations for Iceberg catalog initiali
Fluss supports all Iceberg-compatible catalog types:
**Built-in Catalog Types:**
-- `hive` - Hive Metastore catalog
+- `hive` - Hive Metastore catalog (see [Hive Metastore](../datalake-catalogs/hive.md))
- `hadoop` - Hadoop catalog
-- `rest` - REST catalog
-- `glue` - AWS Glue catalog
+- `rest` - REST catalog (see [Lakekeeper](../datalake-catalogs/lakekeeper.md) and [Polaris](../datalake-catalogs/polaris.md))
+- `glue` - AWS Glue catalog (see [AWS Glue](../datalake-catalogs/glue.md))
- `nessie` - Nessie catalog
- `jdbc` - JDBC catalog
diff --git a/website/src/css/custom.css b/website/src/css/custom.css
index 232037d0f..32e4b9377 100644
--- a/website/src/css/custom.css
+++ b/website/src/css/custom.css
@@ -3310,3 +3310,16 @@ button.clean-btn[class*='toggle'] svg {
html[data-theme='light'] .navbar__search {
opacity: 1;
}
+
+/* =========================================================================
+ Mermaid diagrams
+ ========================================================================= */
+
+/* Mermaid sizes node/edge label boxes assuming its own line-height of 1.5,
+ but the site's paragraph line-height (1.8) also applies inside the SVG's
+ HTML labels — multi-line labels overflow and clip at the bottom. Pin the
+ label line-height back to what mermaid measured with. */
+.docusaurus-mermaid-container foreignObject p {
+ line-height: 1.5;
+ margin: 0;
+}