You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "MehulBatra (via GitHub)" <gi...@apache.org> on 2025/10/11 10:29:01 UTC
[PR] Fluss iceberg quickstart guide [fluss]
MehulBatra opened a new pull request, #1800:
URL: https://github.com/apache/fluss/pull/1800
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Brief change log
1. Added flink-iceberg.md for iceberg related fluss quickstart guide using hadoop catalog
2. Modified flink.md to flink-paimon.md for seperation and changed header for the same
3. Added a note explaining that the $lake suffix and metadata tables functionality (like
$lake$snapshots) is currently in development and referenced issue #1559.
4. Added both the files in quickstart section
5. Added hadoop jar support in flink-dist plugins.xml for hadoop catalog
### Tests
Verified the guide by running manually on my local
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2432494015
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,497 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `lib` directory and download the required Hadoop jar file:
+
+```shell
Review Comment:
use `wget -O lib/hadoop-apache-3.3.5-2.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar`
not `wget -O lib/https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425140418
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
Review Comment:
I really don't want to add hadoop uber lib into `plugins/iceberg` for:
1. It's not always to require hadooop libs for iceberg, for example, if using rest catalog + s3, hadoop is not required. Bundle hadooop libs will make fluss dist large which I really want to avoid. So, it's suggested to add the hadooop libs munually.
2. Actaully, it's not hadoop2 uber libs, it should use hadoop3 libs. See my comment in [here](https://github.com/apache/fluss/pull/1759#discussion_r2422371303)
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
Review Comment:
avro-1.12.0.jar is not needed, see my comment in https://github.com/apache/fluss/pull/1759#discussion_r2422371303
##########
website/docs/intro.md:
##########
@@ -34,7 +34,7 @@ The following is a list of (but not limited to) use-cases that Fluss shines ✨:
## Where to go Next?
-- [QuickStart](quickstart/flink.md): Get started with Fluss in minutes.
+- [QuickStart](quickstart/flink-paimon.md): Get started with Fluss in minutes.
Review Comment:
Also update
```
<Link
className={clsx("hero_button button button--primary button--lg", styles.buttonWidth)}
to="/docs/quickstart/flink">
Quick Start
</Link>
```
to
```
<Link
className={clsx("hero_button button button--primary button--lg", styles.buttonWidth)}
to="/docs/quickstart/flink-paimon">
Quick Start
</Link>
```
in `index.tsx`
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
Review Comment:
We can add jars required by iceberg dynamically
```
volumes:
- ./lib:/tmp/lib
entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
```
See https://github.com/luoyuxia/fluss-iceberg/tree/main/fluss-iceberg for reference
##########
website/docs/intro.md:
##########
@@ -34,7 +34,7 @@ The following is a list of (but not limited to) use-cases that Fluss shines ✨:
## Where to go Next?
-- [QuickStart](quickstart/flink.md): Get started with Fluss in minutes.
+- [QuickStart](quickstart/flink-paimon.md): Get started with Fluss in minutes.
Review Comment:
I also found other reference need to be changed. Seach `quickstart/flink.md` global, and then replace it with `quickstart/flink-paimon.md`
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
+:::
+
+Wait for the checkpoints (~30s) to complete to ensure data is tiered to Iceberg. Then run the following SQL to do real-time analytics:
Review Comment:
`checkpoints` is out of date now after we refactor the lakehouse implementation.
```suggestion
Wait for the configured `datalake.freshness` (~30s) to complete to ensure data is tiered to Iceberg. Then run the following SQL to do real-time analytics:
```
Also remember to modify the statement in paimon page
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
Review Comment:
why iceberg-aws-bundle is required since it don't involve with S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #1800:
URL: https://github.com/apache/fluss/pull/1800#issuecomment-3396748539
> @MehulBatra Thanks for update. 1: The ci fail 2: Also, I notice in the descrition [#1727 (comment)](https://github.com/apache/fluss/issues/1727#issue-3434213735), it's suggested paimon to be a tab, iceberg to be another tab instead of pages. Is it possible to make them as tabs. I think tabs will be more unified to avoid introduce two many pages.
If make them as tabs is not easy, I don't mind change them to tabs in another pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2427854982
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,497 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `lib` directory and download the required Hadoop jar file:
+
+```shell
Review Comment:
use less commands:
```
mkdir lib
wget -O lib/hadoop-apache-3.3.5-2.jar https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425778542
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -57,6 +57,8 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
Review Comment:
We need a step to guide user to create `lib` directory and download [hadoop-apache-3.3.5-2.jar](https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #1800:
URL: https://github.com/apache/fluss/pull/1800#issuecomment-3397809575
somehow it's passing in my local but failing here 😓
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425919266
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -57,6 +57,8 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
Review Comment:
I will add a step guide to add additonal jars, but
I was thinking to add hadoop in the docker image itself for the quickstart? what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425919266
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -57,6 +57,8 @@ services:
datalake.iceberg.warehouse: /tmp/iceberg
volumes:
- shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
Review Comment:
I will add a step guide to add additonal jars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425342880
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
Review Comment:
Sure will address, will remove from dist. Do we add this step in doc to add hadoop jar seperately, or can we just add it seperately in the quick start flink-iceberg image that we create?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2427844468
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,497 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `lib` directory and download the required Hadoop jar file:
+
+```shell
+mkdir lib
+cd lib
+wget https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+cd ..
+```
+
+This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+
+:::info
+The `lib` directory serves as a staging area for additional jars needed by the Fluss coordinator server. The docker-compose configuration (see step 3) mounts this directory and copies all jars to `/opt/fluss/plugins/iceberg/` inside the coordinator container at startup.
+
+You can add more jars to this `lib` directory based on your requirements:
+- **Cloud storage support**: For AWS S3 integration with Iceberg, add the corresponding Iceberg bundle jars (e.g., `iceberg-aws-bundle`)
+- **Custom Hadoop configurations**: Add jars for specific HDFS distributions or custom authentication mechanisms
+- **Other catalog backends**: Add jars needed for alternative Iceberg catalog implementations (e.g., Rest, Hive, Glue)
+
+Any jar placed in the `lib` directory will be automatically loaded by the Fluss coordinator server, making it available for Iceberg integration.
+:::
+
+3. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
+ entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases) and [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/alibaba/fluss/issues/1559) for more details.
Review Comment:
The issue link references 'github.com/alibaba/fluss' but should reference 'github.com/apache/fluss' based on the repository structure mentioned in the PR description.
```suggestion
Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425344742
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
Review Comment:
saw comment will add hadoop jar 3 to address avro concern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #1800:
URL: https://github.com/apache/fluss/pull/1800#issuecomment-3396509861
Thank you @luoyuxia for the review and feedback, addressed your comments, please have a look!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2422717612
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/alibaba/fluss/issues/1559) for more details.
Review Comment:
The issue link references 'github.com/alibaba/fluss' but based on the PR metadata, this should be 'github.com/apache/fluss' to match the correct repository.
```suggestion
Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
```
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
+ <unpack>false</unpack>
+ <useProjectArtifact>false</useProjectArtifact>
+ <useProjectAttachments>false</useProjectAttachments>
+ <useTransitiveDependencies>true</useTransitiveDependencies>
+ <useTransitiveFiltering>true</useTransitiveFiltering>
+ <includes>
+ <include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
+ </includes>
+ </dependencySet>
Review Comment:
This dependency set configuration is identical to the existing one above it (lines 33-41). Consider extracting this into a reusable configuration or add a comment explaining why the same dependency is needed in both plugins/paimon and plugins/iceberg directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425446591
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
Review Comment:
Make sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425450853
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
Review Comment:
Agreed, will make adjustments!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2432595838
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
+:::
+
+Wait for the checkpoints (~30s) to complete to ensure data is tiered to Iceberg. Then run the following SQL to do real-time analytics:
Review Comment:
I changed it in last commit then i saw the flink.md it was named as checkpoint so to maintain the style i wrote it that way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #1800:
URL: https://github.com/apache/fluss/pull/1800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425359424
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
Review Comment:
I think we can add a step to guide users to add hadoop jar. It's not a complex step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425344317
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
Review Comment:
will remove that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2432503713
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
+:::
+
+Wait for the checkpoints (~30s) to complete to ensure data is tiered to Iceberg. Then run the following SQL to do real-time analytics:
Review Comment:
Please don't forget my this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2432599372
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
+Support for querying Iceberg data directly via the `$lake` suffix and metadata tables (like `$lake$snapshots`) is currently in development. See [issue #1559](https://github.com/apache/fluss/issues/1559) for more details.
+:::
+
+Wait for the checkpoints (~30s) to complete to ensure data is tiered to Iceberg. Then run the following SQL to do real-time analytics:
Review Comment:
Didn't miss :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2427859200
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,497 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `lib` directory and download the required Hadoop jar file:
+
+```shell
+mkdir lib
+cd lib
+wget https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+cd ..
+```
+
+This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+
+:::info
+The `lib` directory serves as a staging area for additional jars needed by the Fluss coordinator server. The docker-compose configuration (see step 3) mounts this directory and copies all jars to `/opt/fluss/plugins/iceberg/` inside the coordinator container at startup.
+
+You can add more jars to this `lib` directory based on your requirements:
+- **Cloud storage support**: For AWS S3 integration with Iceberg, add the corresponding Iceberg bundle jars (e.g., `iceberg-aws-bundle`)
+- **Custom Hadoop configurations**: Add jars for specific HDFS distributions or custom authentication mechanisms
+- **Other catalog backends**: Add jars needed for alternative Iceberg catalog implementations (e.g., Rest, Hive, Glue)
+
+Any jar placed in the `lib` directory will be automatically loaded by the Fluss coordinator server, making it available for Iceberg integration.
+:::
+
+3. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
+ entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases) and [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
Review Comment:
Let's wait #1803 merged to update the pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #1800:
URL: https://github.com/apache/fluss/pull/1800#issuecomment-3400016649
@MehulBatra I revert to refer to `flink.md`, then every thing works well. I suggest to still use `flink.md` since in the following pr, we will just have one page `flink.md`, but two tabls for paimon and iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2427948065
##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,497 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `lib` directory and download the required Hadoop jar file:
+
+```shell
+mkdir lib
+cd lib
+wget https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+cd ..
+```
+
+This jar file provides Hadoop 3.3.5 dependencies required for Iceberg's Hadoop catalog integration.
+
+:::info
+The `lib` directory serves as a staging area for additional jars needed by the Fluss coordinator server. The docker-compose configuration (see step 3) mounts this directory and copies all jars to `/opt/fluss/plugins/iceberg/` inside the coordinator container at startup.
+
+You can add more jars to this `lib` directory based on your requirements:
+- **Cloud storage support**: For AWS S3 integration with Iceberg, add the corresponding Iceberg bundle jars (e.g., `iceberg-aws-bundle`)
+- **Custom Hadoop configurations**: Add jars for specific HDFS distributions or custom authentication mechanisms
+- **Other catalog backends**: Add jars needed for alternative Iceberg catalog implementations (e.g., Rest, Hive, Glue)
+
+Any jar placed in the `lib` directory will be automatically loaded by the Fluss coordinator server, making it available for Iceberg integration.
+:::
+
+3. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+ zookeeper:
+ restart: always
+ image: zookeeper:3.9.2
+
+ coordinator-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: coordinatorServer
+ depends_on:
+ - zookeeper
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://coordinator-server:9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+ - ./lib:/tmp/lib
+ entrypoint: ["sh", "-c", "cp -v /tmp/lib/*.jar /opt/fluss/plugins/iceberg/ && exec /docker-entrypoint.sh coordinatorServer"]
+
+ tablet-server:
+ image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+ command: tabletServer
+ depends_on:
+ - coordinator-server
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ bind.listeners: FLUSS://tablet-server:9123
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ kv.snapshot.interval: 0s
+ datalake.format: iceberg
+ datalake.iceberg.type: hadoop
+ datalake.iceberg.warehouse: /tmp/iceberg
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ jobmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ ports:
+ - "8083:8081"
+ command: jobmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+ taskmanager:
+ image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+ depends_on:
+ - jobmanager
+ command: taskmanager
+ environment:
+ - |
+ FLINK_PROPERTIES=
+ jobmanager.rpc.address: jobmanager
+ taskmanager.numberOfTaskSlots: 10
+ taskmanager.memory.process.size: 2048m
+ taskmanager.memory.framework.off-heap.size: 256m
+ volumes:
+ - shared-tmpfs:/tmp/iceberg
+
+volumes:
+ shared-tmpfs:
+ driver: local
+ driver_opts:
+ type: "tmpfs"
+ device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases) and [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/) and then put them to `FLINK_HOME/lib/`.
+- All the following commands involving `docker compose` should be executed in the created working directory that contains the `docker-compose.yml` file.
+:::
+
+Congratulations, you are all set!
+
+## Enter into SQL-Client
+First, use the following command to enter the Flink SQL CLI Container:
+```shell
+docker compose exec jobmanager ./sql-client
+```
+
+**Note**:
+To simplify this guide, three temporary tables have been pre-created with `faker` connector to generate data.
+You can view their schemas by running the following commands:
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_customer;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_order;
+```
+
+```sql title="Flink SQL"
+SHOW CREATE TABLE source_nation;
+```
+
+## Create Fluss Tables
+### Create Fluss Catalog
+Use the following SQL to create a Fluss catalog:
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = 'coordinator-server:9123'
+);
+```
+
+```sql title="Flink SQL"
+USE CATALOG fluss_catalog;
+```
+
+:::info
+By default, catalog configurations are not persisted across Flink SQL client sessions.
+For further information how to store catalog configurations, see [Flink's Catalog Store](https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/catalogs/#catalog-store).
+:::
+
+### Create Tables
+Running the following SQL to create Fluss tables to be used in this guide:
+```sql title="Flink SQL"
+CREATE TABLE fluss_order (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `ptime` AS PROCTIME(),
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_customer (
+ `cust_key` INT NOT NULL,
+ `name` STRING,
+ `phone` STRING,
+ `nation_key` INT NOT NULL,
+ `acctbal` DECIMAL(15, 2),
+ `mktsegment` STRING,
+ PRIMARY KEY (`cust_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE fluss_nation (
+ `nation_key` INT NOT NULL,
+ `name` STRING,
+ PRIMARY KEY (`nation_key`) NOT ENFORCED
+);
+```
+
+```sql title="Flink SQL"
+CREATE TABLE enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+);
+```
+
+## Streaming into Fluss
+
+First, run the following SQL to sync data from source tables to Fluss tables:
+```sql title="Flink SQL"
+EXECUTE STATEMENT SET
+BEGIN
+ INSERT INTO fluss_nation SELECT * FROM `default_catalog`.`default_database`.source_nation;
+ INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
+ INSERT INTO fluss_order SELECT * FROM `default_catalog`.`default_database`.source_order;
+END;
+```
+
+Fluss primary-key tables support high QPS point lookup queries on primary keys. Performing a [lookup join](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/joins/#lookup-join) is really efficient and you can use it to enrich
+the `fluss_orders` table with information from the `fluss_customer` and `fluss_nation` primary-key tables.
+
+```sql title="Flink SQL"
+INSERT INTO enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+
+## Run Ad-hoc Queries on Fluss Tables
+You can now perform real-time analytics directly on Fluss tables.
+For instance, to calculate the number of orders placed by a specific customer, you can execute the following SQL query to obtain instant, real-time results.
+
+```sql title="Flink SQL"
+-- use tableau result mode
+SET 'sql-client.execution.result-mode' = 'tableau';
+```
+
+```sql title="Flink SQL"
+-- switch to batch mode
+SET 'execution.runtime-mode' = 'batch';
+```
+
+```sql title="Flink SQL"
+-- use limit to query the enriched_orders table
+SELECT * FROM enriched_orders LIMIT 2;
+```
+
+**Sample Output**
+```
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| order_key | cust_key | total_price | order_date | order_priority | clerk | cust_name | cust_phone | cust_acctbal | cust_mktsegment | nation_name |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+| 23199744 | 9 | 266.44 | 2024-08-29 | high | Clerk1 | Joe King | 908.207.8513 | 124.28 | FURNITURE | JORDAN |
+| 10715776 | 2 | 924.43 | 2024-11-04 | medium | Clerk3 | Rita Booke | (925) 775-0717 | 172.39 | FURNITURE | UNITED |
++-----------+----------+-------------+------------+----------------+--------+------------+----------------+--------------+-----------------+-------------+
+```
+If you are interested in a specific customer, you can retrieve their details by performing a lookup on the `cust_key`.
+
+```sql title="Flink SQL"
+-- lookup by primary key
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | Al K. Seltzer | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+**Note:** Overall the query results are returned really fast, as Fluss enables efficient primary key lookups for tables with defined primary keys.
+
+## Update/Delete rows on Fluss Tables
+
+You can use `UPDATE` and `DELETE` statements to update/delete rows on Fluss tables.
+### Update
+```sql title="Flink SQL"
+-- update by primary key
+UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
+```
+Then you can `lookup` the specific row:
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+**Sample Output**
+```shell
++----------+---------------+--------------+------------+---------+------------+
+| cust_key | name | phone | nation_key | acctbal | mktsegment |
++----------+---------------+--------------+------------+---------+------------+
+| 1 | fluss_updated | 817-617-7960 | 1 | 533.41 | AUTOMOBILE |
++----------+---------------+--------------+------------+---------+------------+
+```
+Notice that the `name` column has been updated to `fluss_updated`.
+
+### Delete
+```sql title="Flink SQL"
+DELETE FROM fluss_customer WHERE `cust_key` = 1;
+```
+The following SQL query should return an empty result.
+```sql title="Flink SQL"
+SELECT * FROM fluss_customer WHERE `cust_key` = 1;
+```
+
+## Integrate with Iceberg
+### Start the Lakehouse Tiering Service
+To integrate with [Apache Iceberg](https://iceberg.apache.org/), you need to start the `Lakehouse Tiering Service`.
+Open a new terminal, navigate to the `fluss-quickstart-flink-iceberg` directory, and execute the following command within this directory to start the service:
+```shell
+docker compose exec jobmanager \
+ /opt/flink/bin/flink run \
+ /opt/flink/opt/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers coordinator-server:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hadoop \
+ --datalake.iceberg.warehouse /tmp/iceberg
+```
+You should see a Flink Job to tier data from Fluss to Iceberg running in the [Flink Web UI](http://localhost:8083/).
+
+### Streaming into Fluss datalake-enabled tables
+
+By default, tables are created with data lake integration disabled, meaning the Lakehouse Tiering Service will not tier the table's data to the data lake.
+
+To enable lakehouse functionality as a tiered storage solution for a table, you must create the table with the configuration option `table.datalake.enabled = true`.
+Return to the `SQL client` and execute the following SQL statement to create a table with data lake integration enabled:
+```sql title="Flink SQL"
+CREATE TABLE datalake_enriched_orders (
+ `order_key` BIGINT,
+ `cust_key` INT NOT NULL,
+ `total_price` DECIMAL(15, 2),
+ `order_date` DATE,
+ `order_priority` STRING,
+ `clerk` STRING,
+ `cust_name` STRING,
+ `cust_phone` STRING,
+ `cust_acctbal` DECIMAL(15, 2),
+ `cust_mktsegment` STRING,
+ `nation_name` STRING,
+ PRIMARY KEY (`order_key`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Next, perform streaming data writing into the **datalake-enabled** table, `datalake_enriched_orders`:
+```sql title="Flink SQL"
+-- switch to streaming mode
+SET 'execution.runtime-mode' = 'streaming';
+```
+
+```sql title="Flink SQL"
+-- insert tuples into datalake_enriched_orders
+INSERT INTO datalake_enriched_orders
+SELECT o.order_key,
+ o.cust_key,
+ o.total_price,
+ o.order_date,
+ o.order_priority,
+ o.clerk,
+ c.name,
+ c.phone,
+ c.acctbal,
+ c.mktsegment,
+ n.name
+FROM fluss_order o
+ LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS `c`
+ ON o.cust_key = c.cust_key
+ LEFT JOIN fluss_nation FOR SYSTEM_TIME AS OF `o`.`ptime` AS `n`
+ ON c.nation_key = n.nation_key;
+```
+
+### Real-Time Analytics on Fluss datalake-enabled Tables
+
+The data for the `datalake_enriched_orders` table is stored in Fluss (for real-time data) and Iceberg (for historical data).
+
+When querying the `datalake_enriched_orders` table, Fluss uses a union operation that combines data from both Fluss and Iceberg to provide a complete result set -- combines **real-time** and **historical** data.
+
+:::note
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on PR #1800:
URL: https://github.com/apache/fluss/pull/1800#issuecomment-3397024033
> > @MehulBatra Thanks for update. 1: The ci fail 2: Also, I notice in the descrition [#1727 (comment)](https://github.com/apache/fluss/issues/1727#issue-3434213735), it's suggested paimon to be a tab, iceberg to be another tab instead of pages. Is it possible to make them as tabs. I think tabs will be more unified to avoid introduce two many pages.
>
> If make them as tabs is not easy, I don't mind change them to tabs in another pr.
I think we can take that as a seperate PR as it will introduce more changes, I am thinking to get this merged before flink forward session
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fluss iceberg quickstart guide [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425359424
##########
fluss-dist/src/main/assemblies/plugins.xml:
##########
@@ -39,6 +39,17 @@
<include>org.apache.flink:flink-shaded-hadoop-2-uber</include>
</includes>
</dependencySet>
+ <dependencySet>
+ <outputDirectory>plugins/iceberg</outputDirectory>
Review Comment:
I think we can add a step to guide users to add hadoop jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org