You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by gi...@apache.org on 2026/10/21 07:15:38 UTC
(amoro) branch asf-site updated: Regenerated Amoro latest docs page based on 6af75bbe23b79974900997b85f9629d7c1623311 commit
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/amoro.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f17795257 Regenerated Amoro latest docs page based on 6af75bbe23b79974900997b85f9629d7c1623311 commit
f17795257 is described below
commit f177952576a3749fb4f1d99c25f4bef7a8d22625
Author: GitHub Actions <ac...@users.noreply.github.com>
AuthorDate: Wed Oct 21 07:15:35 2026 +0000
Regenerated Amoro latest docs page based on 6af75bbe23b79974900997b85f9629d7c1623311 commit
---
.../docs/latest/flink-getting-started/index.html | 11 +-
output/docs/latest/iceberg-format/index.html | 28 +-
output/docs/latest/index.html | 351 ++++++++++++++++++++-
output/docs/latest/index.xml | 235 +++++++++++++-
output/docs/latest/managing-catalogs/index.html | 40 +++
output/docs/latest/managing-optimizers/index.html | 27 +-
output/docs/latest/search.json | 2 +-
output/docs/latest/sitemap.xml | 80 ++++-
8 files changed, 749 insertions(+), 25 deletions(-)
diff --git a/output/docs/latest/flink-getting-started/index.html b/output/docs/latest/flink-getting-started/index.html
index 3fa591fe9..97c4470ab 100644
--- a/output/docs/latest/flink-getting-started/index.html
+++ b/output/docs/latest/flink-getting-started/index.html
@@ -703,15 +703,16 @@ for more information.</p>
<p><code>./mvnw clean package -pl ':amoro-mixed-flink-runtime-1.18' -am -DskipTests</code></p>
<p>The Flink Runtime Jar is located in the <code>amoro-format-mixed/amoro-mixed-flink/v1.18/amoro-mixed-flink-runtime-1.18/target</code> directory.</p>
<h2 id="environment-preparation">Environment preparation</h2>
+<p>Use Hadoop 3 client libraries; Hadoop 2 is no longer supported. Remove any existing Hadoop 2 uber JAR from Flink’s <code>lib</code> directory before adding the Hadoop 3 JARs.</p>
<p>Download Flink and related dependencies, and download Flink 1.18/1.19/1.20 as needed. Taking Flink 1.18 as an example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#75715e"># Replace version value with the latest Amoro version if needed</span>
</span></span><span style="display:flex;"><span>AMORO_VERSION<span style="color:#f92672">=</span>0.9.0-incubating
</span></span><span style="display:flex;"><span>FLINK_VERSION<span style="color:#f92672">=</span>1.18.1
</span></span><span style="display:flex;"><span>FLINK_MAJOR_VERSION<span style="color:#f92672">=</span>1.18
-</span></span><span style="display:flex;"><span>FLINK_HADOOP_SHADE_VERSION<span style="color:#f92672">=</span>2.7.5
+</span></span><span style="display:flex;"><span>HADOOP_VERSION<span style="color:#f92672">=</span>3.4.3
</span></span><span style="display:flex;"><span>APACHE_FLINK_URL<span style="color:#f92672">=</span>archive.apache.org/dist/flink
</span></span><span style="display:flex;"><span>MAVEN_URL<span style="color:#f92672">=</span>https://repo1.maven.org/maven2
-</span></span><span style="display:flex;"><span>FLINK_CONNECTOR_URL<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>MAVEN_URL<span style="color:#e6db74">}</span>/org/apache/flink
+</span></span><span style="display:flex;"><span>HADOOP_URL<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>MAVEN_URL<span style="color:#e6db74">}</span>/org/apache/hadoop
</span></span><span style="display:flex;"><span>AMORO_CONNECTOR_URL<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>MAVEN_URL<span style="color:#e6db74">}</span>/org/apache/amoro
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Download FLink binary package</span>
@@ -721,12 +722,13 @@ for more information.</p>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>cd flink-<span style="color:#e6db74">${</span>FLINK_VERSION<span style="color:#e6db74">}</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Download Flink Hadoop dependency</span>
-</span></span><span style="display:flex;"><span>wget <span style="color:#e6db74">${</span>FLINK_CONNECTOR_URL<span style="color:#e6db74">}</span>/flink-shaded-hadoop-2-uber/<span style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span style="color:#e6db74">}</span>-10.0/flink-shaded-hadoop-2-uber-<span style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span style="color:#e6db74">}</span>-10.0.jar
+</span></span><span style="display:flex;"><span>wget <span style="color:#e6db74">${</span>HADOOP_URL<span style="color:#e6db74">}</span>/hadoop-client-api/<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>/hadoop-client-api-<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>.jar
+</span></span><span style="display:flex;"><span>wget <span style="color:#e6db74">${</span>HADOOP_URL<span style="color:#e6db74">}</span>/hadoop-client-runtime/<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>/hadoop-client-runtime-<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>.jar
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Download Flink Amoro Connector</span>
</span></span><span style="display:flex;"><span>wget <span style="color:#e6db74">${</span>AMORO_CONNECTOR_URL<span style="color:#e6db74">}</span>/amoro-mixed-format-flink-runtime-<span style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span style="color:#e6db74">}</span>/<span style="color:#e6db74">${</span>AMORO_VERSION<span style="color:#e6db74">}</span>/amoro-mixed-format-flink-runtime-<span style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span style="color:#e6db74">}</span>-<span s [...]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Copy the necessary JAR files to the lib directory</span>
-</span></span><span style="display:flex;"><span>mv flink-shaded-hadoop-2-uber-<span style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span style="color:#e6db74">}</span>-10.0.jar lib
+</span></span><span style="display:flex;"><span>mv hadoop-client-api-<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>.jar hadoop-client-runtime-<span style="color:#e6db74">${</span>HADOOP_VERSION<span style="color:#e6db74">}</span>.jar lib
</span></span><span style="display:flex;"><span>mv amoro-mixed-format-flink-runtime-<span style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span style="color:#e6db74">}</span>-<span style="color:#e6db74">${</span>AMORO_VERSION<span style="color:#e6db74">}</span>.jar lib
</span></span></code></pre></div><p>Modify Flink related configuration files:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>cd flink-1.18.1
@@ -741,7 +743,6 @@ for more information.</p>
</span></span><span style="display:flex;"><span>cp examples/table/ChangelogSocketExample.jar lib
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>cp ../amoro-mixed-flink-runtime-<span style="color:#e6db74">${</span>FLINK_MAJOR_VERSION<span style="color:#e6db74">}</span>-<span style="color:#e6db74">${</span>AMORO_VERSION<span style="color:#e6db74">}</span>.jar lib
-</span></span><span style="display:flex;"><span>cp ../flink-shaded-hadoop-2-uber-<span style="color:#e6db74">${</span>FLINK_HADOOP_SHADE_VERSION<span style="color:#e6db74">}</span>-10.0.jar lib
</span></span></code></pre></div><h3 id="mixed-hive-format">Mixed-Hive format</h3>
<p>Starting from Amoro version 0.3.1, Mixed-Hive format is supported, and data in Amoro Mixed-Hive format tables can be read/written through Flink. When operating on Mixed-Hive format tables through Flink, the following points should be noted:</p>
<ol>
diff --git a/output/docs/latest/iceberg-format/index.html b/output/docs/latest/iceberg-format/index.html
index 74649bb00..41af4556f 100644
--- a/output/docs/latest/iceberg-format/index.html
+++ b/output/docs/latest/iceberg-format/index.html
@@ -686,7 +686,23 @@ However, as data and delete files are written, the read performance and availabi
<img src="../images/formats/iceberg_format.png" alt="Iceberg Format" width="70%" height="70%">
<p>Starting from Amoro v0.4, Iceberg format including v1 and v2 is supported. Users only need to register Iceberg’s catalog in Amoro to host the table for Amoro maintenance. For detailed operation steps, please refer to <a href="../managing-catalogs/">Managing Catalogs</a>.
Amoro maintains the performance and economic availability of Iceberg tables with minimal read/write costs through means such as small file merging, eq-delete file conversion to pos-delete files, duplicate data elimination, and file cleaning, and Amoro has no intrusive impact on the functionality of Iceberg.</p>
-<p>Iceberg format has full upward and downward compatibility features, and in general, users do not have to worry about the compatibility of the Iceberg version used by the engine client with the Iceberg version on which Amoro depends.</p>
+<h3 id="format-version-compatibility">Format version compatibility</h3>
+<p>Amoro automatic maintenance supports Iceberg format versions 1 and 2. Format version 3
+is not supported by the bundled Iceberg writer: a metadata commit can omit fields required
+by newer V3 readers and make the table inaccessible to those readers.</p>
+<p>Amoro skips optimizing evaluation and planning for V3 tables and rejects their maintenance
+and optimizing commits. This also applies to a table upgraded while a transaction is committing.
+Recovery checks the table version before resuming optimizing jobs. If the catalog lookup fails,
+recovery continues with a warning; the writer still checks compatibility before committing.
+Snapshot expiration, dangling delete cleanup, data expiration, automatic tags, and orphan file
+cleanup are all blocked for unsupported tables.</p>
+<p>The compatibility guard does not block catalog discovery or metadata reads and does not
+change the table’s maintenance settings. Read-only table summary collection remains available
+when optimizing is disabled and table summary collection is enabled. Loading and reading still
+depend on the features supported by the bundled Iceberg reader. The table name, unsupported
+version, and supported versions are reported in AMS logs; rejected maintenance and optimizing
+processes closed by the guard also use the existing failure or closure reporting. This protection
+does not repair metadata already written without required V3 fields.</p>
<p>Amoro supports all catalog types supported by Iceberg, including but not limited to: Hadoop, Hive, Glue, JDBC, Nessie, Snowflake, and so on.</p>
<p>Amoro supports all storage types supported by Iceberg, including but not limited to: Hadoop, S3, AliyunOSS, GCS, ECS, and so on.</p>
</div>
@@ -708,7 +724,15 @@ Amoro maintains the performance and economic availability of Iceberg tables with
<div id="toc" class="markdown-body">
<div id="full">
- <nav id="TableOfContents"></nav>
+ <nav id="TableOfContents">
+ <ul>
+ <li>
+ <ul>
+ <li><a href="#format-version-compatibility">Format version compatibility</a></li>
+ </ul>
+ </li>
+ </ul>
+</nav>
</div>
</div>
diff --git a/output/docs/latest/index.html b/output/docs/latest/index.html
index 21f4561a9..2bbc9581f 100644
--- a/output/docs/latest/index.html
+++ b/output/docs/latest/index.html
@@ -32,14 +32,13 @@
<!DOCTYPE html>
<html>
<head>
- <meta name="generator" content="Hugo 0.166.0">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
- <title>Amoro</title>
+ <title>Introduction</title>
@@ -84,6 +83,7 @@
+ <meta name="generator" content="Hugo 0.168.0">
</head>
<body>
@@ -389,7 +389,7 @@
<li>
- <a class="top-menu" id="" href="./">
+ <a class="top-menu" id="active" href="./">
<span>Introduction</span>
</a>
</li>
@@ -399,70 +399,253 @@
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Concepts">
<span>Concepts</span>
</a>
</li>
+ <div id="Concepts" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./catalogs/">Catalogs</a>
+ </li>
+
+ <li>
+ <a href="./self-optimizing/">Self-Optimizing</a>
+ </li>
+
+ <li>
+ <a href="./table-watermark/">Table Watermark</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Configuration">
<span>Configuration</span>
</a>
</li>
+ <div id="Configuration" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./ams-config/">AMS Configuration</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Admin_Guides">
<span>Admin Guides</span>
</a>
</li>
+ <div id="Admin_Guides" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./deployment/">Deployment</a>
+ </li>
+
+ <li>
+ <a href="./deployment-on-kubernetes/">Deployment On Kubernetes</a>
+ </li>
+
+ <li>
+ <a href="./managing-catalogs/">Managing Catalogs</a>
+ </li>
+
+ <li>
+ <a href="./managing-optimizers/">Managing Optimizers</a>
+ </li>
+
+ <li>
+ <a href="./using-customized-encryption-method/">Using Customized Encryption Method for Configurations</a>
+ </li>
+
+ <li>
+ <a href="./using-kyuubi/">Using Kyuubi By Terminal</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#User_Guides">
<span>User Guides</span>
</a>
</li>
+ <div id="User_Guides" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./using-tables/">Using Tables</a>
+ </li>
+
+ <li>
+ <a href="./configurations/">Configurations</a>
+ </li>
+
+ <li>
+ <a href="./cdc-ingestion/">CDC Ingestion</a>
+ </li>
+
+ <li>
+ <a href="./metrics/">Metrics</a>
+ </li>
+
+ <li>
+ <a href="./rest-api/">REST API</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Formats">
<span>Formats</span>
</a>
</li>
+ <div id="Formats" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./formats-overview/">Overview</a>
+ </li>
+
+ <li>
+ <a href="./iceberg-format/">Iceberg</a>
+ </li>
+
+ <li>
+ <a href="./paimon-format/">Paimon</a>
+ </li>
+
+ <li>
+ <a href="./mixed-iceberg-format/">Mixed-Iceberg</a>
+ </li>
+
+ <li>
+ <a href="./mixed-hive-format/">Mixed-Hive</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Flink">
<span>Flink</span>
</a>
</li>
+ <div id="Flink" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./flink-getting-started/">Flink Getting Started</a>
+ </li>
+
+ <li>
+ <a href="./flink-ddl/">Flink DDL</a>
+ </li>
+
+ <li>
+ <a href="./flink-dml/">Flink DML</a>
+ </li>
+
+ <li>
+ <a href="./flink-cdc-ingestion/">Flink CDC Ingestion</a>
+ </li>
+
+ <li>
+ <a href="./flink-datastream/">Flink DataStream</a>
+ </li>
+
+ <li>
+ <a href="./flink-using-logstore/">Using Logstore</a>
+ </li>
+
+ </ul>
+ </div>
<li>
- <a class="top-menu" id="" href="">
+
+
+ <a class="top-menu chevron-toggle collapsed" data-toggle="collapse" href="#Spark">
<span>Spark</span>
</a>
</li>
+ <div id="Spark" class="collapse in">
+ <ul class="sub-menu">
+
+ <li>
+ <a href="./spark-getting-started/">Spark Getting Started</a>
+ </li>
+
+ <li>
+ <a href="./spark-configuration/">Spark Configuration</a>
+ </li>
+
+ <li>
+ <a href="./spark-ddl/">Spark DDL</a>
+ </li>
+
+ <li>
+ <a href="./spark-queries/">Spark Queries</a>
+ </li>
+
+ <li>
+ <a href="./spark-writes/">Spark Writes</a>
+ </li>
+
+ </ul>
+ </div>
+
+
+
+
+
+ <li>
+ <a class="top-menu" id="" href="./trino/">
+ <span>Trino</span>
+ </a>
+ </li>
</ul>
@@ -482,7 +665,132 @@
<a href="https://amoro.apache.org/docs/latest/../../docs/0.8.1/"><strong>0.8.1-incubating</strong></a>.
</p>
</div>
- </div>
+ <!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+<h1 id="introduction">Introduction</h1>
+<p>Amoro is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.</p>
+<h2 id="architecture">Architecture</h2>
+<p>The architecture of Amoro is as follows:</p>
+<p align="center">
+<img src="images/introduce_amoro.png" alt="Architecture" width="90%" height="90%">
+</p>
+<p>The core components of Amoro include:</p>
+<ul>
+<li>AMS: Amoro Management Service provides Lakehouse management features, like self-optimizing, data expiration, etc.
+It also provides a unified catalog service for all compute engines, which can also be combined with existing metadata services.</li>
+<li>Plugins: Amoro provides a wide selection of external plugins to meet different scenarios.
+<ul>
+<li>Optimizers: The self-optimizing execution engine plugin asynchronously performs merging, sorting, deduplication,
+layout optimization, and other operations on all type table format tables.</li>
+<li>Terminal: SQL command-line tools, provide various implementations like local Spark and Kyuubi.</li>
+<li>LogStore: Provide millisecond to second level SLAs for real-time data processing based on message queues like Kafka and Pulsar.</li>
+</ul>
+</li>
+</ul>
+<h2 id="supported-table-formats">Supported table formats</h2>
+<p>Amoro can manage tables of different table formats, similar to how MySQL/ClickHouse can choose different storage engines.
+Amoro meets diverse user needs by using different table formats. Currently, Amoro supports three table formats:</p>
+<ul>
+<li>Iceberg format: means using the native table format of the Apache Iceberg, which has all the features and characteristics of Iceberg.</li>
+<li>Mixed-Iceberg format: built on top of Iceberg format, which can accelerate data processing using LogStore
+and provides more efficient query performance and streaming read capability in CDC scenarios.</li>
+<li>Mixed-Hive format: has the same features as the Mixed-Iceberg tables but is compatible with a Hive table.
+Support upgrading Hive tables to Mixed-Hive tables, and allow Hive’s native read and write methods after upgrading.</li>
+</ul>
+<h2 id="supported-engines">Supported engines</h2>
+<h3 id="iceberg-format">Iceberg format</h3>
+<p>Iceberg format tables use the engine integration method provided by the Iceberg community.
+For details, please refer to: <a href="https://iceberg.apache.org/docs/latest/">Iceberg Docs</a>.</p>
+<h3 id="paimon-format">Paimon format</h3>
+<p>Paimon format tables use the engine integration method provided by the Paimon community.
+For details, please refer to: <a href="https://paimon.apache.org/docs/master/">Paimon Docs</a>.</p>
+<h3 id="mixed-format">Mixed format</h3>
+<p>Amoro support multiple processing engines for Mixed format as below:</p>
+<table>
+ <thead>
+ <tr>
+ <th>Processing Engine</th>
+ <th>Version</th>
+ <th>Batch Read</th>
+ <th>Batch Write</th>
+ <th>Batch Overwrite</th>
+ <th>Streaming Read</th>
+ <th>Streaming Write</th>
+ <th>Create Table</th>
+ <th>Alter Table</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Flink</td>
+ <td>1.18.x, 1.19.x, 1.20.x</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ </tr>
+ <tr>
+ <td>Spark</td>
+ <td>3.3, 3.4, 3.5</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Hive</td>
+ <td>2.x, 3.x</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Trino</td>
+ <td>406</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ </tbody>
+</table>
+<h2 id="user-cases">User cases</h2>
+<h3 id="self-managed-streaming-lakehouse">Self-managed streaming Lakehouse</h3>
+<p>Amoro makes it easier for users to handle the challenges of writing to a real-time data lake, such as ingesting append-only event logs or CDC data from databases.
+In these scenarios, the rapid increase of fragment and redundant files cannot be ignored.
+To address this issue, Amoro provides a pluggable streaming data self-optimizing mechanism that automatically compacts fragment files and removes expired data, ensuring high-quality table queries while reducing system costs.</p>
+<h3 id="stream-and-batch-fused-data-pipeline">Stream-and-batch-fused data pipeline</h3>
+<p>Whether in the AI or BI business field , the requirement for real-time analysis is becoming increasingly high. The traditional approach of using one streaming job to complete all data processing from the source to the end is no longer applicable. There is an increasing demand for layered construction of streaming data pipeline, and the traditional layered construction approach based on message queues can cause a inconsistency problem between the streaming and batch data processing. Bu [...]
+<h3 id="cloud-native-lakehouse">Cloud-native Lakehouse</h3>
+<p>Currently, most data platforms and products are tightly coupled with their underlying infrastructure(such as the storage layer). The migration of infrastructure, such as switching to cloud-native OSS, may require extensive adaptation efforts or even be impossible. However, Amoro provides an infra-decoupled, lake-native architecture built on top of the infrastructure. This allows products based on Amoro to interact with the underlying infrastructure through a unified interface (Amoro C [...]
+</div>
@@ -501,7 +809,26 @@
<div id="toc" class="markdown-body">
<div id="full">
- <nav id="TableOfContents"></nav>
+ <nav id="TableOfContents">
+ <ul>
+ <li><a href="#architecture">Architecture</a></li>
+ <li><a href="#supported-table-formats">Supported table formats</a></li>
+ <li><a href="#supported-engines">Supported engines</a>
+ <ul>
+ <li><a href="#iceberg-format">Iceberg format</a></li>
+ <li><a href="#paimon-format">Paimon format</a></li>
+ <li><a href="#mixed-format">Mixed format</a></li>
+ </ul>
+ </li>
+ <li><a href="#user-cases">User cases</a>
+ <ul>
+ <li><a href="#self-managed-streaming-lakehouse">Self-managed streaming Lakehouse</a></li>
+ <li><a href="#stream-and-batch-fused-data-pipeline">Stream-and-batch-fused data pipeline</a></li>
+ <li><a href="#cloud-native-lakehouse">Cloud-native Lakehouse</a></li>
+ </ul>
+ </li>
+ </ul>
+</nav>
</div>
</div>
diff --git a/output/docs/latest/index.xml b/output/docs/latest/index.xml
index 5e773c706..9574a8e31 100644
--- a/output/docs/latest/index.xml
+++ b/output/docs/latest/index.xml
@@ -1,9 +1,9 @@
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
- <title>Amoro</title>
+ <title>Introduction on Amoro</title>
<link>https://amoro.apache.org/docs/latest/</link>
- <description>Recent content on Amoro</description>
+ <description>Recent content in Introduction on Amoro</description>
<generator>Hugo</generator>
<language>en-us</language>
@@ -11,5 +11,236 @@
<atom:link href="https://amoro.apache.org/docs/latest/index.xml" rel="self" type="application/rss+xml" />
+ <item>
+ <title></title>
+ <link>https://amoro.apache.org/docs/latest/readme/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/readme/</guid>
+ <description><h2 id="amoro-docs">Amoro Docs</h2>
<p>This directory contains the documentation content of Amoro.</p>
<ul>
<li>The documentation is written in Markdown format.</li>
<li>The images referenced in the documentation are saved in the <code>images</code> folder.</li>
</ul>
<h3 id="style">Style</h3>
<ul>
<li>Proper nouns should start w [...]
+ </item>
+ <item>
+ <title>AMS Configuration</title>
+ <link>https://amoro.apache.org/docs/latest/ams-config/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/ams-config/</guid>
+ <description><!--
 ~ Licensed to the Apache Software Foundation (ASF) under one
 ~ or more contributor license agreements. See the NOTICE file
 ~ distributed with this work for additional information
 ~ regarding copyright ownership. The ASF licenses this file
 ~ to you under the Apache License, Version 2.0 (the
 ~ "License"); you may not use this file except in compliance
 ~ with the License. You may obtain a copy of the License at&#x [...]
+ </item>
+ <item>
+ <title>Catalogs</title>
+ <link>https://amoro.apache.org/docs/latest/catalogs/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/catalogs/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>CDC Ingestion</title>
+ <link>https://amoro.apache.org/docs/latest/cdc-ingestion/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/cdc-ingestion/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Configurations</title>
+ <link>https://amoro.apache.org/docs/latest/configurations/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/configurations/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Deployment</title>
+ <link>https://amoro.apache.org/docs/latest/deployment/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/deployment/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Deployment On Kubernetes</title>
+ <link>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink CDC Ingestion</title>
+ <link>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DataStream</title>
+ <link>https://amoro.apache.org/docs/latest/flink-datastream/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-datastream/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DDL</title>
+ <link>https://amoro.apache.org/docs/latest/flink-ddl/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-ddl/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink DML</title>
+ <link>https://amoro.apache.org/docs/latest/flink-dml/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-dml/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Flink Getting Started</title>
+ <link>https://amoro.apache.org/docs/latest/flink-getting-started/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-getting-started/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Iceberg</title>
+ <link>https://amoro.apache.org/docs/latest/iceberg-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/iceberg-format/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Managing Catalogs</title>
+ <link>https://amoro.apache.org/docs/latest/managing-catalogs/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/managing-catalogs/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Managing Optimizers</title>
+ <link>https://amoro.apache.org/docs/latest/managing-optimizers/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/managing-optimizers/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Metrics</title>
+ <link>https://amoro.apache.org/docs/latest/metrics/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/metrics/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Mixed-Hive</title>
+ <link>https://amoro.apache.org/docs/latest/mixed-hive-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/mixed-hive-format/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Mixed-Iceberg</title>
+ <link>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Overview</title>
+ <link>https://amoro.apache.org/docs/latest/formats-overview/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/formats-overview/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Paimon</title>
+ <link>https://amoro.apache.org/docs/latest/paimon-format/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/paimon-format/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>REST API</title>
+ <link>https://amoro.apache.org/docs/latest/rest-api/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/rest-api/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Self-Optimizing</title>
+ <link>https://amoro.apache.org/docs/latest/self-optimizing/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/self-optimizing/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Configuration</title>
+ <link>https://amoro.apache.org/docs/latest/spark-configuration/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/spark-configuration/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark DDL</title>
+ <link>https://amoro.apache.org/docs/latest/spark-ddl/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/spark-ddl/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Getting Started</title>
+ <link>https://amoro.apache.org/docs/latest/spark-getting-started/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/spark-getting-started/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Queries</title>
+ <link>https://amoro.apache.org/docs/latest/spark-queries/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/spark-queries/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Spark Writes</title>
+ <link>https://amoro.apache.org/docs/latest/spark-writes/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/spark-writes/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Table Watermark</title>
+ <link>https://amoro.apache.org/docs/latest/table-watermark/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/table-watermark/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Trino</title>
+ <link>https://amoro.apache.org/docs/latest/trino/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/trino/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Customized Encryption Method for Configurations</title>
+ <link>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Kyuubi By Terminal</title>
+ <link>https://amoro.apache.org/docs/latest/using-kyuubi/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/using-kyuubi/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Logstore</title>
+ <link>https://amoro.apache.org/docs/latest/flink-using-logstore/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/flink-using-logstore/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
+ <item>
+ <title>Using Tables</title>
+ <link>https://amoro.apache.org/docs/latest/using-tables/</link>
+ <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
+ <guid>https://amoro.apache.org/docs/latest/using-tables/</guid>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - h [...]
+ </item>
</channel>
</rss>
diff --git a/output/docs/latest/managing-catalogs/index.html b/output/docs/latest/managing-catalogs/index.html
index 65f551ab8..e5790fbe0 100644
--- a/output/docs/latest/managing-catalogs/index.html
+++ b/output/docs/latest/managing-catalogs/index.html
@@ -726,6 +726,46 @@ please read the following guidelines and select the appropriate creation accordi
</ul>
<h3 id="configure-table-properties">Configure table properties</h3>
<p>If you want to add the same table properties to all tables under a catalog, you can add these table properties here on the catalog level. If you also configure this property on the table level, the property on the table will take effect.</p>
+<div class="info">
+ When configuring table-level properties on a catalog, you <strong>must prefix the key with <code>table.</code></strong>. The prefix will be automatically stripped when the property is applied to tables. For example, to set the default <code>self-optimizing.group</code> for all tables under the catalog, you need to configure <code>table.self-optimizing.group</code> instead of <code>self-optimizing.group</code>. Properties without the <code>table.</code> prefix will not be recognized a [...]
+</div>
+<p>The following table lists some examples. For the full list of available table properties, see <a href="../user-guides/configurations.md">Table configurations</a>.</p>
+<table>
+ <thead>
+ <tr>
+ <th>Catalog property key</th>
+ <th>Effective table property key</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>table.self-optimizing.enabled</code></td>
+ <td><code>self-optimizing.enabled</code></td>
+ <td>Enable/disable self-optimizing for all tables</td>
+ </tr>
+ <tr>
+ <td><code>table.self-optimizing.group</code></td>
+ <td><code>self-optimizing.group</code></td>
+ <td>Set the default optimizer group for all tables</td>
+ </tr>
+ <tr>
+ <td><code>table.table-expire.enabled</code></td>
+ <td><code>table-expire.enabled</code></td>
+ <td>Enable/disable table expiration for all tables</td>
+ </tr>
+ <tr>
+ <td><code>table.clean-orphan-file.enabled</code></td>
+ <td><code>clean-orphan-file.enabled</code></td>
+ <td>Enable/disable orphan file cleaning for all tables</td>
+ </tr>
+ <tr>
+ <td><code>table.log-store.enabled</code></td>
+ <td><code>log-store.enabled</code></td>
+ <td>Enable/disable LogStore for all tables</td>
+ </tr>
+ </tbody>
+</table>
<h2 id="rest-catalog">REST Catalog</h2>
<p>When a user needs to create a Iceberg REST Catalog, they can choose <strong>External Catalog Type</strong>、<strong>Custom Metastore Type</strong>、<strong>Iceberg Table Format</strong>, configure properties include:
<strong>catalog-impl=org.apache.iceberg.rest.RESTCatalog</strong>, <strong>uri=$restCatalog_uri</strong>.</p>
diff --git a/output/docs/latest/managing-optimizers/index.html b/output/docs/latest/managing-optimizers/index.html
index d02c34f00..4e9f57af7 100644
--- a/output/docs/latest/managing-optimizers/index.html
+++ b/output/docs/latest/managing-optimizers/index.html
@@ -1206,7 +1206,7 @@ contain several optimizers with the same container implementation to facilitate
<td>All</td>
<td>No</td>
<td>false</td>
- <td>Whether to enable dynamic resource allocation (AIP-5) for this group: the group then grows its optimizer count automatically in response to optimizing demand, bounded by <code>dynamic-allocation.max-parallelism</code>. Not supported on externally-registered optimizers. Manual optimizer operations (scale-out/release via the dashboard) are not recommended on groups with dynamic allocation enabled.</td>
+ <td>Whether to enable dynamic resource allocation for this group: the group then grows its optimizer count automatically in response to optimizing demand, bounded by <code>dynamic-allocation.max-parallelism</code>. Not supported on externally-registered optimizers. Manual optimizer operations (scale-out/release via the dashboard) are not recommended on groups with dynamic allocation enabled.</td>
</tr>
<tr>
<td>dynamic-allocation.min-parallelism</td>
@@ -1296,7 +1296,29 @@ contain several optimizers with the same container implementation to facilitate
<li>When using <code>shutdown-timeout-ms</code> with a Flink Optimizer, also raise <code>flink-conf.task.cancellation.timeout</code> (default 180000) accordingly — the graceful drain on job cancellation is capped below Flink’s cancellation watchdog, which otherwise fails the whole TaskManager.</li>
</ul>
</div>
-<h3 id="edit-optimizer-group">Edit optimizer group</h3>
+<h3 id="dynamic-resource-allocation">Dynamic resource allocation</h3>
+<p>Dynamic resource allocation (DRA) lets a group grow and shrink its optimizer count automatically based on optimizing demand, instead of running a fixed number of optimizers. Enable it by setting <code>dynamic-allocation.enabled</code> to <code>true</code> on the group; once enabled, <code>dynamic-allocation.min-parallelism</code> and <code>dynamic-allocation.max-parallelism</code> take over from the deprecated flat <code>min-parallelism</code>.</p>
+<p><strong>Scale-up</strong>: when optimizing demand (queued tasks) persists for <code>dynamic-allocation.scheduler-backlog-timeout</code>, the group scales out for the first time. While demand keeps persisting, it scales out again every <code>dynamic-allocation.sustained-backlog-timeout</code>, up to <code>dynamic-allocation.max-parallelism</code>.</p>
+<p><strong>Scale-down</strong>: an optimizer that has been idle for <code>dynamic-allocation.executor-idle-timeout</code> becomes a scale-down candidate. At most one optimizer is removed per evaluation round, and only while the group has no scale-up demand, with at least <code>dynamic-allocation.scale-down-cooldown</code> between removals. A scaled-down optimizer is drained gracefully — it stops receiving new tasks and is removed once its in-flight tasks finish; if draining exceeds <code [...]
+<div class="info">
+ <ul>
+<li>DRA is not supported for externally-registered optimizers.</li>
+<li>Avoid manual scale-out/release from the dashboard on a group with DRA enabled — it fights with the auto scaler.</li>
+<li><code>dynamic-allocation.min-parallelism</code> and <code>dynamic-allocation.max-parallelism</code> must both be reachable as multiples of <code>dynamic-allocation.executor-parallelism</code>, the number of threads added or removed per scaling step.</li>
+</ul>
+
+</div>
+<p>An example enabling DRA on a Kubernetes optimizer group, entered in the group’s <code>properties</code>:</p>
+<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.enabled</span>: <span style="color:#66d9ef">true</span> <span style="color:#75715e"># Turn on dynamic resource allocation</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.min-parallelism</span>: <span style="color:#ae81ff">8</span> <span style="color:#75715e"># Keep at least 1 optimizer (executor-parallelism=8) running</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.max-parallelism</span>: <span style="color:#ae81ff">64</span> <span style="color:#75715e"># Scale out to at most 8 optimizers</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.executor-parallelism</span>: <span style="color:#ae81ff">8</span> <span style="color:#75715e"># Threads per optimizer pod; 4-8 is recommended on Kubernetes</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.scheduler-backlog-timeout</span>: <span style="color:#ae81ff">1min </span> <span style="color:#75715e"># Wait 1 min of sustained backlog before the first scale-out</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.sustained-backlog-timeout</span>: <span style="color:#ae81ff">30s </span> <span style="color:#75715e"># Re-evaluate and scale out every 30s while backlog persists</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.executor-idle-timeout</span>: <span style="color:#ae81ff">10min </span> <span style="color:#75715e"># Consider an optimizer idle after 10 min without tasks</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.scale-down-cooldown</span>: <span style="color:#ae81ff">1min </span> <span style="color:#75715e"># Wait at least 1 min between scale-down removals</span>
+</span></span><span style="display:flex;"><span><span style="color:#f92672">dynamic-allocation.drain-timeout</span>: <span style="color:#ae81ff">15min </span> <span style="color:#75715e"># Force-remove a draining optimizer after 15 min</span>
+</span></span></code></pre></div><h3 id="edit-optimizer-group">Edit optimizer group</h3>
<p>You can click the <code>edit</code> button on the <code>Optimizer Groups</code> page to modify the configuration of the Optimizer group.</p>
<h3 id="remove-optimizer-group">Remove optimizer group</h3>
<p>You can click the <code>remove</code> button on the <code>Optimizer Groups</code> page to delete the optimizer group, but only if the group is
@@ -1514,6 +1536,7 @@ If the optimizer runs normally, you will see an optimizer with the status <code>
<li><a href="#optimizer-group">Optimizer group</a>
<ul>
<li><a href="#add-optimizer-group">Add optimizer group</a></li>
+ <li><a href="#dynamic-resource-allocation">Dynamic resource allocation</a></li>
<li><a href="#edit-optimizer-group">Edit optimizer group</a></li>
<li><a href="#remove-optimizer-group">Remove optimizer group</a></li>
</ul>
diff --git a/output/docs/latest/search.json b/output/docs/latest/search.json
index 0637a088a..fa5aaa7c6 100644
--- a/output/docs/latest/search.json
+++ b/output/docs/latest/search.json
@@ -1 +1 @@
-[]
\ No newline at end of file
+[{"categories":null,"content":"Amoro Docs This directory contains the documentation content of Amoro.\nThe documentation is written in Markdown format. The images referenced in the documentation are saved in the images folder. Style Proper nouns should start with a capital letter, like Hadoop、Hive、Iceberg、Amoro ","description":"","title":"","uri":"https://amoro.apache.org/docs/latest/readme/"},{"categories":null,"content":" AMS Configuration Amoro Management Service Configuration The con [...]
\ No newline at end of file
diff --git a/output/docs/latest/sitemap.xml b/output/docs/latest/sitemap.xml
index 3ecb0371f..54c7a1740 100644
--- a/output/docs/latest/sitemap.xml
+++ b/output/docs/latest/sitemap.xml
@@ -2,10 +2,88 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
- <loc>https://amoro.apache.org/docs/latest/</loc>
+ <loc>https://amoro.apache.org/docs/latest/readme/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/admin-guides/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/ams-config/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/catalogs/</loc>
</url><url>
<loc>https://amoro.apache.org/docs/latest/categories/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/cdc-ingestion/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/concepts/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/configuration/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/configurations/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/deployment/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/engines/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-datastream/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-ddl/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-dml/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-getting-started/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/formats/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/iceberg-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/managing-catalogs/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/managing-optimizers/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/metrics/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/mixed-hive-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/formats-overview/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/paimon-format/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/rest-api/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/self-optimizing/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-configuration/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-ddl/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-getting-started/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-queries/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/spark-writes/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/table-watermark/</loc>
</url><url>
<loc>https://amoro.apache.org/docs/latest/tags/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/trino/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/user-guides/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/using-kyuubi/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/flink-using-logstore/</loc>
+ </url><url>
+ <loc>https://amoro.apache.org/docs/latest/using-tables/</loc>
</url>
</urlset>