You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@fluss.apache.org by ab...@apache.org on 2026/09/14 14:26:07 UTC
(fluss) branch main updated: [docs] Add HTTP Gateway lakehouse quickstart (#4231)
This is an automated email from the ASF dual-hosted git repository.
fresh-borzoni pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fluss.git
The following commit(s) were added to refs/heads/main by this push:
new 1a33b3312 [docs] Add HTTP Gateway lakehouse quickstart (#4231)
1a33b3312 is described below
commit 1a33b331255be9961d1e5e102288a21f1f5be9c1
Author: Pranav Shukla <pr...@gmail.com>
AuthorDate: Mon Sep 14 19:56:01 2026 +0530
[docs] Add HTTP Gateway lakehouse quickstart (#4231)
* docs: add HTTP Gateway lakehouse quickstart
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* test: validate gateway quickstart environment
* [docs] Add HTTP Gateway lakehouse quickstart
* changes to incorporate minor fixes for issue-4221-http-gateway-quickstart
* changes to incorporate minor fixes for issue-4221-http-gateway-quickstart
* [docs] Update HTTP Gateway lakehouse quickstart
* Updated gateway-lakehouse.md
* [docs] Update HTTP Gateway lakehouse quickstart- added images and minor fix
---
website/docs/assets/gateway-lakehouse-dark.png | Bin 0 -> 134700 bytes
website/docs/assets/gateway-lakehouse-light.png | Bin 0 -> 443298 bytes
website/docs/gateway/index.md | 3 +
website/docs/quickstart/gateway-lakehouse.md | 540 ++++++++++++++++++++++++
4 files changed, 543 insertions(+)
diff --git a/website/docs/assets/gateway-lakehouse-dark.png b/website/docs/assets/gateway-lakehouse-dark.png
new file mode 100644
index 000000000..072ea5311
Binary files /dev/null and b/website/docs/assets/gateway-lakehouse-dark.png differ
diff --git a/website/docs/assets/gateway-lakehouse-light.png b/website/docs/assets/gateway-lakehouse-light.png
new file mode 100644
index 000000000..bfa0812cc
Binary files /dev/null and b/website/docs/assets/gateway-lakehouse-light.png differ
diff --git a/website/docs/gateway/index.md b/website/docs/gateway/index.md
index 548e4b86a..d4b06f18e 100644
--- a/website/docs/gateway/index.md
+++ b/website/docs/gateway/index.md
@@ -16,6 +16,9 @@ Fluss Gateway is a stateless REST service for metadata, DDL, and schema-aware
batch writes. Any Gateway instance can handle any request, so instances can be
scaled behind a load balancer.
+For an end-to-end walkthrough — ingesting events via the Gateway and querying
+ them through a Paimon lakehouse — see
+[Ingest HTTP Events into a Real-Time Lakehouse](../quickstart/gateway-lakehouse.md).
To run the Gateway as a binary distribution or container, see
[Deploying Fluss Gateway](../install-deploy/deploying-gateway.md).
diff --git a/website/docs/quickstart/gateway-lakehouse.md b/website/docs/quickstart/gateway-lakehouse.md
new file mode 100644
index 000000000..73409da26
--- /dev/null
+++ b/website/docs/quickstart/gateway-lakehouse.md
@@ -0,0 +1,540 @@
+---
+title: Ingesting HTTP Events into a Real-Time Lakehouse
+sidebar_position: 5
+---
+import ThemedImage from '@site/src/components/ThemedImage';
+
+This guide shows how an application can send JSON events to Fluss through
+the Gateway REST API and immediately read them back — without setting up
+a message queue, a Flink ingestion pipeline, or a separate Paimon writer.
+The same table serves both real-time queries and historical lakehouse
+analytics, with Paimon tiering handled automatically in the background.
+
+The example tracks orders flowing through a simple order management system.
+The application writes one HTTP request per batch, no Kafka producer, no
+Flink ingestion job, no separate Paimon writer. Fluss provides immediate
+query visibility in the hot tier, and the reusable Lakehouse Tiering Service
+handles writing to Paimon continuously in the background.
+
+In this quickstart, you will:
+
+<ThemedImage
+ alt="gateway lakehouse guide"
+ light="gateway-lakehouse-light.png"
+ dark="gateway-lakehouse-dark.png"
+/>
+
+- spin up Fluss, Fluss Gateway, Flink, Paimon, and RustFS with Docker Compose;
+- create a lake-enabled table through the Gateway REST API;
+- write JSON order records over HTTP;
+- read the records back immediately through Flink SQL before any tiering occurs;
+- confirm that the rows are continuously tiered into Paimon.
+
+:::caution Developer Preview - Fluss 1.0 (unreleased)
+Fluss Gateway is a preview feature in Fluss 1.0, which has not yet been
+released. This quickstart uses its unauthenticated `trust` mode and is
+intended only for local development. See
+[Fluss Gateway security](/docs/gateway/index.md#security) before planning
+a production deployment.
+
+No pre-built Docker Hub image is available yet – this guide requires
+building the Gateway image from source (see
+[Build the Gateway image](#build-the-gateway-image) below). The Gateway
+does not yet support reading records; this guide reads data back through
+Flink SQL.
+:::
+
+## Environment Setup
+
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/)
+and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are
+installed on your machine.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/)
+(however, Compose v1 might work with a few adaptions).
+:::
+
+### Build the Gateway image
+
+:::note Fluss v1.0 developer preview
+This guide covers a feature shipping in the upcoming Fluss 1.0 release.
+A pre-built Docker Hub image will be available at GA - until then,
+building it locally takes one extra step: run the script below from the
+root of your [Fluss source repository](https://github.com/apache/fluss)
+:::
+Run this from the root of your Fluss source checkout:
+```shell
+docker/fluss-gateway/build.sh
+```
+
+This compiles the Gateway inside a `rust:1.88-bookworm` builder container
+(no local Rust toolchain needed) and produces a local image tagged
+`fluss-gateway:dev`, which the Compose file below references directly. The
+first build may take several minutes.
+
+### Starting required components
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-gateway-lakehouse
+cd fluss-quickstart-gateway-lakehouse
+```
+
+2. Create a `lib` directory and download the Paimon S3 plugin jar required
+ by the Fluss servers:
+
+```shell
+mkdir lib
+curl -fL -o "lib/paimon-s3-$PAIMON_VERSION$.jar" \
+ "https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-s3/$PAIMON_VERSION$/paimon-s3-$PAIMON_VERSION$.jar"
+```
+
+:::info
+The `apache/fluss-quickstart-flink` image already includes the Flink-side
+dependencies used by this guide. Only the Fluss server-side `paimon-s3`
+plugin still needs to be downloaded and mounted into the Fluss containers.
+:::
+
+3. Create a `docker-compose.yml` file with the following content:
+
+```yaml
+services:
+ #begin RustFS (S3-compatible storage)
+ rustfs:
+ image: rustfs/rustfs:1.0.0-alpha.83
+ ports:
+ - "9000:9000"
+ - "9001:9001"
+ environment:
+ - RUSTFS_ACCESS_KEY=rustfsadmin
+ - RUSTFS_SECRET_KEY=rustfsadmin
+ - RUSTFS_CONSOLE_ENABLE=true
+ volumes:
+ - rustfs-data:/data
+ command: /data
+ rustfs-init:
+ image: minio/mc
+ depends_on:
+ - rustfs
+ entrypoint: >
+ /bin/sh -c "
+ until mc alias set rustfs http://rustfs:9000 rustfsadmin rustfsadmin; do
+ echo 'Waiting for RustFS...';
+ sleep 1;
+ done;
+ mc mb --ignore-existing rustfs/fluss;
+ "
+ #end
+ coordinator-server:
+ image: apache/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ zookeeper:
+ condition: service_started
+ rustfs-init:
+ condition: service_completed_successfully
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: s3://fluss/remote-data
+ s3.endpoint: http://rustfs:9000
+ s3.access-key: rustfsadmin
+ s3.secret-key: rustfsadmin
+ s3.region: us-east-1
+ s3.path-style-access: true
+ s3.assumed.role.arn: arn:aws:iam::000000000000:role/rustfsadmin
+ s3.assumed.role.sts.endpoint: http://rustfs:9000
+ datalake.enabled: true
+ datalake.format: paimon
+ datalake.paimon.metastore: filesystem
+ datalake.paimon.warehouse: s3://fluss/paimon
+ datalake.paimon.s3.endpoint: http://rustfs:9000
+ datalake.paimon.s3.access-key: rustfsadmin
+ datalake.paimon.s3.secret-key: rustfsadmin
+ datalake.paimon.s3.path.style.access: true
+ volumes:
+ - ./lib/paimon-s3-$PAIMON_VERSION$.jar:/opt/fluss/plugins/paimon/paimon-s3-$PAIMON_VERSION$.jar
+ tablet-server:
+ image: apache/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: s3://fluss/remote-data
+ s3.endpoint: http://rustfs:9000
+ s3.access-key: rustfsadmin
+ s3.secret-key: rustfsadmin
+ s3.region: us-east-1
+ s3.path-style-access: true
+ s3.assumed.role.arn: arn:aws:iam::000000000000:role/rustfsadmin
+ s3.assumed.role.sts.endpoint: http://rustfs:9000
+ datalake.enabled: true
+ datalake.format: paimon
+ datalake.paimon.metastore: filesystem
+ datalake.paimon.warehouse: s3://fluss/paimon
+ datalake.paimon.s3.endpoint: http://rustfs:9000
+ datalake.paimon.s3.access-key: rustfsadmin
+ datalake.paimon.s3.secret-key: rustfsadmin
+ datalake.paimon.s3.path.style.access: true
+ volumes:
+ - ./lib/paimon-s3-$PAIMON_VERSION$.jar:/opt/fluss/plugins/paimon/paimon-s3-$PAIMON_VERSION$.jar
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+ #begin Fluss Gateway
+ gateway:
+ image: fluss-gateway:dev
+ depends_on:
+ - coordinator-server
+ ports:
+ - "8080:8080"
+ environment:
+ - FLUSS_GATEWAY__CLUSTER__DEFAULT__BOOTSTRAP__SERVERS=coordinator-server:9123
+ #end
+ jobmanager:
+ image: apache/fluss-quickstart-flink:$FLUSS_QUICKSTART_FLINK_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ entrypoint: ["/opt/flink/init_paimon.sh"]
+ command: ["jobmanager"]
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager:
+ image: apache/fluss-quickstart-flink:$FLUSS_QUICKSTART_FLINK_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ entrypoint: ["/opt/flink/init_paimon.sh"]
+ command: ["taskmanager"]
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.task.off-heap.size: 128m
+ sql-client:
+ image: apache/fluss-quickstart-flink:$FLUSS_QUICKSTART_FLINK_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ entrypoint: ["/opt/flink/init_paimon.sh"]
+ command: ["/opt/sql-client/sql-client"]
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ rest.address: jobmanager
+
+volumes:
+ rustfs-data:
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and
+ a `ZooKeeper` server.
+- **Fluss Gateway:** a stateless REST service used to create tables and
+ write records in this guide, listening on `localhost:8080`.
+- **Flink Cluster**: a Flink `JobManager`, a Flink `TaskManager`, and a
+ Flink SQL client container, used to run the Lakehouse Tiering Service and
+ query results.
+- **RustFS**: an S3-compatible storage system used both as Fluss remote
+ storage and Paimon's filesystem warehouse.
+
+:::tip
+[RustFS](https://github.com/rustfs/rustfs) is used as replacement for S3 in
+this quickstart example, for your production setup you may want to
+configure this to use cloud file system. See [here](/maintenance/tiered-storage/filesystems/overview.md)
+for information on how to setup cloud file systems.
+:::
+
+4. Start all containers.
+
+```shell
+docker compose up -d
+docker compose ps
+```
+
+:::note
+The `sql-client` service may exit after `docker compose up -d` because no
+interactive terminal is attached. This is expected. You will start a new SQL
+client container with `docker compose run --rm sql-client` later in this guide.
+:::
+
+5. Wait until the Gateway can reach Fluss:
+
+```shell
+until curl -fsS \
+ http://localhost:8080/v1/clusters/default/databases >/dev/null; do
+ sleep 2
+done
+```
+
+:::note
+All the following commands involving `docker compose` should be executed in the
+working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Create a datalake-enabled table via the Gateway
+
+Set the endpoint and resource names used in the examples:
+
+```bash
+GATEWAY_URL=http://localhost:8080
+CLUSTER=default
+DATABASE=gateway_demo
+```
+
+### Create a database
+
+```bash
+curl -sS --fail-with-body -X POST \
+ -H 'Content-Type: application/json' \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases" \
+ -d "{\"database\":\"$DATABASE\"}"
+```
+
+### Create the table with lakehouse integration enabled
+
+Set `table.datalake.enabled` and `table.datalake.freshness` in `configs` at
+creation time — the same properties the `lakehouse.md` guide sets via
+`WITH (...)` in Flink SQL:
+
+```bash
+curl -sS --fail-with-body -X POST \
+ -H 'Content-Type: application/json' \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE/tables" \
+ -d '{
+ "table_name": "orders",
+ "columns": [
+ {"name": "order_id", "data_type": {"type": "INTEGER"}, "nullable": false},
+ {"name": "customer", "data_type": {"type": "STRING"}, "nullable": true},
+ {"name": "amount_cents", "data_type": {"type": "BIGINT"}, "nullable": true},
+ {"name": "status", "data_type": {"type": "STRING"}, "nullable": true}
+ ],
+ "primary_key": ["order_id"],
+ "distribution": {"bucket_count": 1, "bucket_keys": ["order_id"]},
+ "configs": {
+ "table.datalake.enabled": "true",
+ "table.datalake.freshness": "30s"
+ }
+ }'
+```
+
+:::note
+`amount_cents` stores the order amount as an integer number of cents, to
+keep the HTTP payload simple.
+:::
+
+### Write records
+
+```bash
+curl -sS --fail-with-body -X POST \
+ -H 'Content-Type: application/json' \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE/tables/orders/records" \
+ -d '{
+ "entries": [
+ {"id": "order-1", "upsert": {"order_id": 1, "customer": "Alice", "amount_cents": 4599, "status": "placed"}},
+ {"id": "order-2", "upsert": {"order_id": 2, "customer": "Bob", "amount_cents": 12000, "status": "placed"}},
+ {"id": "order-3", "upsert": {"order_id": 3, "customer": "Carol", "amount_cents": 750, "status": "placed"}}
+ ]
+ }'
+```
+
+:::note
+`amount_cents` values fit comfortably in a JSON number here. For `BIGINT`
+or `DECIMAL` values that exceed JSON number precision (values beyond 2^53),
+pass them as base-10 strings instead — e.g. `"amount_cents": "9999999999999999"`.
+:::
+
+A successful response looks like:
+
+```json
+{
+ "row_count": 3,
+ "success_count": 3,
+ "error_count": 0,
+ "successes": [{"id": "order-1"}, {"id": "order-2"}, {"id": "order-3"}],
+ "failures": []
+}
+```
+
+## Query Immediately
+
+Open the Flink SQL client:
+
+```shell
+docker compose run --rm sql-client
+```
+Create the Fluss catalog and switch to it:
+
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123',
+ 'paimon.s3.access-key' = 'rustfsadmin',
+ 'paimon.s3.secret-key' = 'rustfsadmin'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+USE gateway_demo;
+SET 'sql-client.execution.result-mode' = 'tableau';
+SET 'execution.runtime-mode' = 'batch';
+```
+Query the table immediately after writing – no need to wait for tiering:
+
+```sql title="Flink SQL"
+SELECT order_id, customer, amount_cents, status FROM orders;
+```
+
+The result includes all three orders even though Paimon has not committed
+its first snapshot yet. Union Read gets the newest rows directly from Fluss
+and combines them with any data already tiered:
+
+```text
++----------+----------+---------------+---------+
+| order_id | customer | amount_cents | status |
++----------+----------+---------------+---------+
+| 1 | Alice | 4599 | placed |
+| 2 | Bob | 12000 | placed |
+| 3 | Carol | 750 | placed |
++----------+----------+---------------+---------+
+```
+
+## Start the Lakehouse Tiering Service
+
+Submit the Lakehouse Tiering Service as a detached Flink job:
+
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run -d \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format paimon \
+ --datalake.paimon.metastore filesystem \
+ --datalake.paimon.warehouse s3://fluss/paimon \
+ --datalake.paimon.s3.endpoint http://rustfs:9000 \
+ --datalake.paimon.s3.access.key rustfsadmin \
+ --datalake.paimon.s3.secret.key rustfsadmin \
+ --datalake.paimon.s3.path.style.access true
+```
+
+The [Flink Web UI](http://localhost:8083/) should show one running tiering job.
+
+## Verify Data in Paimon
+
+After approximately 30 seconds, inspect the Paimon snapshots:
+
+```sql title="Flink SQL"
+SELECT snapshot_id, total_record_count
+FROM orders$lake$snapshots
+ORDER BY snapshot_id;
+```
+
+Then check the lake row count:
+
+```sql title="Flink SQL"
+SELECT COUNT(*) AS lake_rows FROM orders$lake;
+```
+
+The count becomes `3` after the tiering cycle commits the rows.
+
+To see the difference between the two query forms, write one more record
+through the Gateway from another terminal. If you're in a fresh terminal,
+re-set the same variables used earlier:
+
+```bash
+GATEWAY_URL=http://localhost:8080
+CLUSTER=default
+DATABASE=gateway_demo
+```
+and then continue with:
+
+```bash
+curl -sS --fail-with-body -X POST \
+ -H 'Content-Type: application/json' \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE/tables/orders/records" \
+ -d '{"entries": [{"id": "order-4", "upsert": {"order_id": 4, "customer": "Dave", "amount_cents": 2200, "status": "placed"}}]}'
+```
+
+Re-run `SELECT ... FROM orders` in the SQL client – `order_id 4` appears
+immediately in the unified view. The lake-only view, `orders$lake`, will
+reflect it once the tiering service has processed it.
+
+The two query forms serve different purposes:
+
+| Query | Data read | Expected freshness |
+|---|---|---|
+| `SELECT ... FROM orders` | Fluss hot data + latest Paimon snapshot | Sub-second |
+| `SELECT ... FROM orders$lake` | Paimon only | ~30s tiering window |
+
+### Quitting the SQL Client
+
+```sql title="Flink SQL"
+exit;
+```
+
+## What This Quickstart Demonstrates
+
+The application in this quickstart never touches Kafka, Flink, or Paimon directly.
+It only makes HTTP requests to the Gateway. Specifically, there is no need for:
+
+- a Kafka topic or producer;
+- a dedicated Flink ingestion job;
+- a Paimon writer;
+- separate table definitions for the streaming and lake layers.
+
+The Lakehouse Tiering Service does run on Flink, but as shared infrastructure
+that serves all Lake-enabled Fluss tables — not as code the application owns
+or operates.
+
+## Preview limitations
+
+- The Gateway's 1.0 preview only implements `trust` mode security — it does
+ not authenticate callers or terminate TLS. Don't expose `$GATEWAY_URL`
+ directly in production; put it behind an authenticated, TLS-terminating
+ ingress.
+- The Gateway does not support record reads, primary-key/prefix lookups, or
+ log scans — this is why the guide reads back through Flink SQL.
+- HTTP 200 from a write request can still contain partial failures; always
+ check both `successes` and `failures` in the response body.
+
+See the full [Fluss Gateway reference](/docs/gateway/index.md) for details.
+
+## Clean up
+
+Exit the SQL client, then delete the table and database through the
+Gateway, then stop the containers:
+
+```bash
+curl -sS --fail-with-body -X DELETE \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE/tables/orders"
+curl -sS --fail-with-body -X DELETE \
+ "$GATEWAY_URL/v1/clusters/$CLUSTER/databases/$DATABASE"
+```
+
+```shell
+docker compose down -v
+```
+
+## Learn more
+
+Now that you're up and running with the Fluss Gateway and a real-time
+lakehouse, check out the [Fluss Gateway reference](/docs/gateway/index.md)
+for the full REST API, or the [Streaming Lakehouse](lakehouse.md) guide for
+the equivalent all-Flink-SQL workflow.