Re: [PR] [doc] documentation for gravitino integration [fluss]

["VladBanar (via GitHub)" <gi...@apache.org>]

VladBanar commented on code in PR #3669:
URL: https://github.com/apache/fluss/pull/3669#discussion_r3721092854


##########
website/docs/streaming-lakehouse/datalake-catalogs/gravitino.md:
##########
@@ -0,0 +1,194 @@
+---
+title: Gravitino
+sidebar_position: 3
+---
+
+# Apache Gravitino
+
+## Introduction
+
+[Apache Gravitino](https://gravitino.apache.org/) is a high-performance metadata management system that provides a unified metadata layer for data lake and lakehouse architectures. It offers an [Iceberg REST Catalog](https://iceberg.apache.org/rest-catalog-spec) interface with credential vending support, making tiered Iceberg tables discoverable and queryable by any Iceberg-compatible engine.
+
+This guide explains how to configure Fluss to use Apache Gravitino as its Iceberg REST catalog. For general Iceberg integration details (table mapping, data types, limitations), see [Iceberg Integration](/docs/next/streaming-lakehouse/datalake-formats/iceberg/).
+
+## How It Works
+
+When Fluss is configured with Gravitino as its Iceberg REST catalog:
+
+1. Fluss creates and manages Iceberg table metadata through Gravitino's REST API
+2. The [tiering service](/docs/next/install-deploy/deploying-streaming-lakehouse/#starting-the-tiering-service) writes data to object storage and commits snapshots via Gravitino
+3. Gravitino provides S3 credentials dynamically through credential vending (no static credentials needed)
+4. Any Iceberg-compatible engine (Flink, Spark, Trino, StarRocks, etc.) can discover and query the tiered tables through Gravitino
+
+## Prerequisites
+
+### Running Gravitino Instance
+
+You need a running Gravitino instance (version 1.3.0+). Refer to the [Gravitino Getting Started](https://gravitino.apache.org/docs/1.3.0/getting-started/index) guide for deployment instructions.
+
+### AWS Bundle for S3 Support
+
+Gravitino requires the AWS bundle for S3 support:
+
+**For Docker deployments:**
+The `apache/gravitino:latest` image already includes the required bundle.
+
+**For native deployments:**
+Download and place the AWS bundle in `iceberg-rest-server/libs/`:
+
+```bash
+wget -P ${GRAVITINO_HOME}/iceberg-rest-server/libs/ \
+    https://repo1.maven.org/maven2/org/apache/gravitino/iceberg-aws-bundle/1.3.0/iceberg-aws-bundle-1.3.0.jar
+```
+
+### Create Gravitino Catalog with Credential Vending
+
+After deploying Gravitino, create a metalake and catalog with credential vending enabled:
+
+```bash
+# Create metalake
+curl -X POST http://<gravitino-host>:8090/api/metalakes \
+    -H "Content-Type: application/json" \
+    -d '{"name": "{metalake name}", "comment": "Fluss metadata lake"}'
+
+# Create Iceberg catalog with credential vending
+curl -X POST http://<gravitino-host>:8090/api/metalakes/{metalake name}/catalogs \
+    -H "Content-Type: application/json" \
+    -d '{
+      "name": "fluss_iceberg_catalog",
+      "type": "RELATIONAL",
+      "provider": "lakehouse-iceberg",
+      "comment": "Iceberg REST catalog for Fluss",
+      "properties": {
+        "uri": "jdbc:postgresql://<postgres-host>:5432/gravitino",
+        "warehouse": "s3://<bucket>/iceberg-data/",
+        "catalog-backend": "jdbc",
+        "jdbc-driver": "org.postgresql.Driver",
+        "jdbc-user": "postgres",
+        "jdbc-password": "password",
+        "s3-endpoint": "http://<s3-endpoint>:9000",
+        "s3-access-key-id": "<access-key>",
+        "s3-secret-access-key": "<secret-key>",
+        "credential-providers": "s3-secret-key"
+      }
+    }'
+```
+
+> **NOTE**: The `credential-providers: s3-secret-key` property enables credential vending. Gravitino will provide temporary S3 credentials to Fluss instead of requiring static credentials in Fluss configuration.
+
+## Configure Fluss with Gravitino
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: rest
+datalake.iceberg.uri: http://<gravitino-host>:9001/iceberg/
+datalake.iceberg.warehouse: fluss_iceberg_catalog
+datalake.iceberg.header.X-Iceberg-Access-Delegation: vended-credentials
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining properties to the Iceberg REST catalog client. You can add any additional [Iceberg REST catalog properties](https://iceberg.apache.org/docs/1.10.1/configuration/#catalog-properties) using the same prefix. For example:
+
+```yaml
+# Optional: pass additional REST catalog properties
+datalake.iceberg.header.Authorization: Bearer <token>
+```
+
+**Key Properties:**
+
+- `datalake.iceberg.uri`: Gravitino's Iceberg REST API endpoint (port 9001)
+- `datalake.iceberg.warehouse`: The catalog name created in Gravitino (not `metalake.catalog`, just `catalog`)
+- `datalake.iceberg.header.X-Iceberg-Access-Delegation: vended-credentials`: Enables credential vending
+
+With credential vending enabled, you do **not** need to specify `s3-access-key-id` or `s3-secret-access-key` in Fluss configuration. However, you still need S3 endpoint configuration:
+
+```yaml
+# S3 endpoint configuration (required)
+datalake.iceberg.s3.endpoint: http://<s3-endpoint>:9000
+datalake.iceberg.s3.path-style-access: true
+datalake.iceberg.s3.region: us-east-1
+```
+
+#### Hadoop Dependencies
+
+Some FileIO implementations require Hadoop classes. Place the pre-bundled Hadoop JAR into `FLUSS_HOME/plugins/iceberg/`:
+
+```bash
+wget -P ${FLUSS_HOME}/plugins/iceberg/ \
+    https://repo1.maven.org/maven2/io/trino/hadoop/hadoop-apache/3.3.5-2/hadoop-apache-3.3.5-2.jar
+```
+
+See [Iceberg - Hadoop Dependencies](/docs/next/streaming-lakehouse/datalake-formats/iceberg/#1-hadoop-dependencies-configuration) for alternative approaches.
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service setup](/docs/next/streaming-lakehouse/datalake-formats/iceberg/#start-tiering-service-to-iceberg) to prepare the required JARs and start the tiering service. Use REST catalog parameters with credential vending when launching the Flink tiering job:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-*.jar \
+    --fluss.bootstrap.servers <coordinator-host>:9123 \
+    --datalake.format iceberg \
+    --datalake.iceberg.type rest \
+    --datalake.iceberg.uri http://<gravitino-host>:9001/iceberg/ \
+    --datalake.iceberg.warehouse fluss_iceberg_catalog \
+    --datalake.iceberg.header.X-Iceberg-Access-Delegation vended-credentials \
+    --datalake.iceberg.s3.endpoint http://<s3-endpoint>:9000 \
+    --datalake.iceberg.s3.path-style-access true
+```
+
+> **NOTE**: Credential vending secures the Fluss-to-Iceberg communication but doesn't affect Flink-to-S3 reads. Flink requires AWS SDK credentials for direct S3 access.
+
+## Metalake Configuration
+
+The metalake is **not** specified in Fluss configuration. Instead, it's configured on the Gravitino server side in `gravitino.conf`:
+
+```properties
+gravitino.iceberg-rest.catalog-config-provider = dynamic-config-provider
+gravitino.iceberg-rest.gravitino-uri = http://localhost:8090
+gravitino.iceberg-rest.gravitino-metalake = {metalake name}
+```
+
+When Fluss requests a catalog by name (e.g., `fluss_iceberg_catalog`), Gravitino's REST service automatically looks it up in the configured metalake (`{metalake name}`). This allows centralized management without requiring clients to know about metalakes.
+
+## Usage Example
+
+### Create a Datalake-Enabled Table
+
+```sql
+USE CATALOG fluss_catalog;
+
+CREATE TABLE orders (
+    `order_id` BIGINT,
+    `customer_id` INT NOT NULL,
+    `total_price` DECIMAL(15, 2),
+    `order_date` DATE,
+    `status` STRING,
+    PRIMARY KEY (`order_id`) NOT ENFORCED
+) WITH (
+    'table.datalake.enabled' = 'true',
+    'table.datalake.freshness' = '30s'
+);
+```
+
+Once the tiering service is running, Fluss automatically creates the corresponding Iceberg table in Gravitino and begins tiering data.
+
+### Query Data
+
+```sql
+SET 'execution.runtime-mode' = 'batch';
+
+-- Union read: combines fresh data in Fluss with historical data in Iceberg
+SELECT COUNT(*) FROM orders;

Review Comment:
   Well it's possible to check in RustFs UI if new parquet files were added but i think querying it would make more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org