Re: [PR] [Docs] Fluss/flink - Iceberg quickstart guide [fluss]

["MehulBatra (via GitHub)" <gi...@apache.org>]

MehulBatra commented on code in PR #1800:
URL: https://github.com/apache/fluss/pull/1800#discussion_r2425446591


##########
website/docs/quickstart/flink-iceberg.md:
##########
@@ -0,0 +1,473 @@
+---
+title: Real-Time Analytics with Flink (Iceberg)
+sidebar_position: 2
+---
+
+# Real-Time Analytics With Flink (Iceberg)
+
+This guide will get you up and running with Apache Flink to do real-time analytics, covering some powerful features of Fluss,
+including integrating with Apache Iceberg.
+The guide is derived from [TPC-H](https://www.tpc.org/tpch/) **Q5**.
+
+For more information on working with Flink, refer to the [Apache Flink Engine](engine-flink/getting-started.md) section.
+
+## Environment Setup
+### Prerequisites
+
+Before proceeding with this guide, ensure that [Docker](https://docs.docker.com/engine/install/) and the [Docker Compose plugin](https://docs.docker.com/compose/install/linux/) are installed on your machine.
+All commands were tested with Docker version 27.4.0 and Docker Compose version v2.30.3.
+
+:::note
+We encourage you to use a recent version of Docker and [Compose v2](https://docs.docker.com/compose/releases/migrate/) (however, Compose v1 might work with a few adaptions).
+:::
+
+### Starting required components
+
+We will use `docker compose` to spin up the required components for this tutorial.
+
+1. Create a working directory for this guide.
+
+```shell
+mkdir fluss-quickstart-flink-iceberg
+cd fluss-quickstart-flink-iceberg
+```
+
+2. Create a `docker-compose.yml` file with the following content:
+
+
+```yaml
+services:
+  zookeeper:
+    restart: always
+    image: zookeeper:3.9.2
+
+  coordinator-server:
+    image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+    command: coordinatorServer
+    depends_on:
+      - zookeeper
+    environment:
+      - |
+        FLUSS_PROPERTIES=
+        zookeeper.address: zookeeper:2181
+        bind.listeners: FLUSS://coordinator-server:9123
+        remote.data.dir: /tmp/fluss/remote-data
+        datalake.format: iceberg
+        datalake.iceberg.type: hadoop
+        datalake.iceberg.warehouse: /tmp/iceberg
+    volumes:
+      - shared-tmpfs:/tmp/iceberg
+
+  tablet-server:
+    image: fluss/fluss:$FLUSS_DOCKER_VERSION$
+    command: tabletServer
+    depends_on:
+      - coordinator-server
+    environment:
+      - |
+        FLUSS_PROPERTIES=
+        zookeeper.address: zookeeper:2181
+        bind.listeners: FLUSS://tablet-server:9123
+        data.dir: /tmp/fluss/data
+        remote.data.dir: /tmp/fluss/remote-data
+        kv.snapshot.interval: 0s
+        datalake.format: iceberg
+        datalake.iceberg.type: hadoop
+        datalake.iceberg.warehouse: /tmp/iceberg
+    volumes:
+      - shared-tmpfs:/tmp/iceberg
+
+  jobmanager:
+    image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+    ports:
+      - "8083:8081"
+    command: jobmanager
+    environment:
+      - |
+        FLINK_PROPERTIES=
+        jobmanager.rpc.address: jobmanager
+    volumes:
+      - shared-tmpfs:/tmp/iceberg
+
+  taskmanager:
+    image: fluss/quickstart-flink:1.20-$FLUSS_DOCKER_VERSION$
+    depends_on:
+      - jobmanager
+    command: taskmanager
+    environment:
+      - |
+        FLINK_PROPERTIES=
+        jobmanager.rpc.address: jobmanager
+        taskmanager.numberOfTaskSlots: 10
+        taskmanager.memory.process.size: 2048m
+        taskmanager.memory.framework.off-heap.size: 256m
+    volumes:
+      - shared-tmpfs:/tmp/iceberg
+
+volumes:
+  shared-tmpfs:
+    driver: local
+    driver_opts:
+      type: "tmpfs"
+      device: "tmpfs"
+```
+
+The Docker Compose environment consists of the following containers:
+- **Fluss Cluster:** a Fluss `CoordinatorServer`, a Fluss `TabletServer` and a `ZooKeeper` server.
+- **Flink Cluster**: a Flink `JobManager` and a Flink `TaskManager` container to execute queries.
+
+**Note:** The `fluss/quickstart-flink` image is based on [flink:1.20.1-java17](https://hub.docker.com/layers/library/flink/1.20-java17/images/sha256:bf1af6406c4f4ad8faa46efe2b3d0a0bf811d1034849c42c1e3484712bc83505) and
+includes the [fluss-flink](engine-flink/getting-started.md), [iceberg-flink](https://iceberg.apache.org/docs/latest/flink/) and
+[flink-connector-faker](https://flink-packages.org/packages/flink-faker) to simplify this guide.
+
+3. To start all containers, run:
+```shell
+docker compose up -d
+```
+This command automatically starts all the containers defined in the Docker Compose configuration in detached mode.
+
+Run
+```shell
+docker container ls -a
+```
+to check whether all containers are running properly.
+
+You can also visit http://localhost:8083/ to see if Flink is running normally.
+
+:::note
+- If you want to additionally use an observability stack, follow one of the provided quickstart guides [here](maintenance/observability/quickstart.md) and then continue with this guide.
+- If you want to run with your own Flink environment, remember to download the [fluss-flink connector jar](/downloads), [flink-connector-faker](https://github.com/knaufk/flink-faker/releases), [iceberg-flink connector jar](https://iceberg.apache.org/docs/latest/flink/), [avro-1.12.0.jar](https://mvnrepository.com/artifact/org.apache.avro/avro/1.12.0), and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) and then put them to `FLINK_HOME/lib/`.

Review Comment:
   Make sense!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org