You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by gi...@apache.org on 2026/01/05 08:15:41 UTC
(amoro) branch asf-site updated: Regenerated Amoro latest docs page based on 88b7ba4ba8a833a3db43823dd319668963aee088 commit
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/amoro.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bdc7b5b4b Regenerated Amoro latest docs page based on 88b7ba4ba8a833a3db43823dd319668963aee088 commit
bdc7b5b4b is described below
commit bdc7b5b4b5995f5bc89e8903cf384dd776300ab0
Author: GitHub Actions <ac...@users.noreply.github.com>
AuthorDate: Mon Jan 5 08:15:37 2026 +0000
Regenerated Amoro latest docs page based on 88b7ba4ba8a833a3db43823dd319668963aee088 commit
---
.../deployment-on-kubernetes/index.html | 1 -
.../docs/latest/admin-guides/deployment/index.html | 1 -
output/docs/latest/admin-guides/index.xml | 12 +-
.../admin-guides/managing-catalogs/index.html | 1 -
.../admin-guides/managing-optimizers/index.html | 1 -
.../using-customized-encryption-method/index.html | 1 -
.../latest/admin-guides/using-kyuubi/index.html | 1 -
output/docs/latest/ams-config/index.html | 856 ++++++++--------
output/docs/latest/concept/catalogs/index.html | 1 -
.../docs/latest/concept/self-optimizing/index.html | 1 -
.../docs/latest/concept/table-watermark/index.html | 1 -
output/docs/latest/concepts/index.xml | 6 +-
.../latest/configuration/ams-config/index.html | 1 -
output/docs/latest/configuration/index.xml | 2 +-
output/docs/latest/configurations/index.html | 794 +++++++--------
.../latest/deployment-on-kubernetes/index.html | 2 +-
output/docs/latest/deployment/index.html | 166 ++--
output/docs/latest/engines/index.xml | 24 +-
output/docs/latest/flink-datastream/index.html | 92 +-
output/docs/latest/flink-ddl/index.html | 534 +++++-----
output/docs/latest/flink-dml/index.html | 766 +++++++--------
output/docs/latest/flink-using-logstore/index.html | 82 +-
output/docs/latest/flink/cdc-ingestion/index.html | 1 -
output/docs/latest/flink/datastream/index.html | 1 -
output/docs/latest/flink/ddl/index.html | 1 -
output/docs/latest/flink/dml/index.html | 1 -
.../docs/latest/flink/getting-started/index.html | 1 -
output/docs/latest/flink/using-logstore/index.html | 1 -
output/docs/latest/formats/iceberg/index.html | 1 -
output/docs/latest/formats/index.xml | 10 +-
output/docs/latest/formats/mixed-hive/index.html | 1 -
.../docs/latest/formats/mixed-iceberg/index.html | 1 -
output/docs/latest/formats/overview/index.html | 1 -
output/docs/latest/formats/paimon/index.html | 1 -
output/docs/latest/index.html | 120 +--
output/docs/latest/index.xml | 66 +-
output/docs/latest/managing-optimizers/index.html | 1030 ++++++++++----------
output/docs/latest/metrics/index.html | 820 ++++++++--------
output/docs/latest/mixed-iceberg-format/index.html | 8 +-
output/docs/latest/search.json | 2 +-
output/docs/latest/self-optimizing/index.html | 72 +-
output/docs/latest/spark-ddl/index.html | 12 +-
output/docs/latest/spark-queries/index.html | 4 +-
output/docs/latest/spark-writes/index.html | 6 +-
output/docs/latest/spark/configuration/index.html | 1 -
output/docs/latest/spark/ddl/index.html | 1 -
.../docs/latest/spark/getting-started/index.html | 1 -
output/docs/latest/spark/queries/index.html | 1 -
output/docs/latest/spark/writes/index.html | 1 -
output/docs/latest/trino/index.html | 144 +--
.../latest/user-guides/cdc-ingestion/index.html | 1 -
.../latest/user-guides/configurations/index.html | 1 -
output/docs/latest/user-guides/index.xml | 10 +-
output/docs/latest/user-guides/metrics/index.html | 1 -
output/docs/latest/user-guides/rest-api/index.html | 1 -
.../latest/user-guides/using-tables/index.html | 1 -
output/docs/latest/using-tables/index.html | 60 +-
57 files changed, 2850 insertions(+), 2881 deletions(-)
diff --git a/output/docs/latest/admin-guides/deployment-on-kubernetes/index.html b/output/docs/latest/admin-guides/deployment-on-kubernetes/index.html
index 81fda9484..ae7df80aa 100644
--- a/output/docs/latest/admin-guides/deployment-on-kubernetes/index.html
+++ b/output/docs/latest/admin-guides/deployment-on-kubernetes/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/deployment-on-kubernetes/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/deployment-on-kubernetes/">
</head>
diff --git a/output/docs/latest/admin-guides/deployment/index.html b/output/docs/latest/admin-guides/deployment/index.html
index b9096ab32..63435c5a2 100644
--- a/output/docs/latest/admin-guides/deployment/index.html
+++ b/output/docs/latest/admin-guides/deployment/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/deployment/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/deployment/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/deployment/">
</head>
diff --git a/output/docs/latest/admin-guides/index.xml b/output/docs/latest/admin-guides/index.xml
index 5cb7b1dfd..3647adc83 100644
--- a/output/docs/latest/admin-guides/index.xml
+++ b/output/docs/latest/admin-guides/index.xml
@@ -12,42 +12,42 @@
<link>https://amoro.apache.org/docs/latest/deployment/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/deployment/</guid>
- <description>Deployment You can choose to download the stable release package from download page, or the source code form Github and compile it according to the README.
System requirements Java 8 is required. Optional: A RDBMS (PostgreSQL 14.x or higher, MySQL 5.5 or higher) Optional: ZooKeeper 3.4.x or higher Download the distribution All released package can be downloaded from download page. You can download apache-amoro-x.y.z-bin.tar.gz (x.y.z is the release number), and you [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Deployment On Kubernetes</title>
<link>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</guid>
- <description>Deploy AMS On Kubernetes Requirements If you want to deploy AMS on Kubernetes, you’d better get a sense of the following things.
Use AMS official docker image or build AMS docker image An active Kubernetes cluster Kubectl Helm3+ Amoro Official Docker Image You can find the official docker image at Amoro Docker Hub.
The following are images that can be used in a production environment.
apache/amoro
This is an image built based on the Amoro binary distrib [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Managing Catalogs</title>
<link>https://amoro.apache.org/docs/latest/managing-catalogs/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/managing-catalogs/</guid>
- <description>Managing Catalogs Users can import your test or online clusters through the catalog management function provided by the AMS Dashboard. Before adding a new Catalog, please read the following guidelines and select the appropriate creation according to your actual needs.
Create catalog In Amoro, the catalog is a namespace for a group of libraries and tables. Under the catalog, it is further divided into different databases, and under each database, there are different [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Managing Optimizers</title>
<link>https://amoro.apache.org/docs/latest/managing-optimizers/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/managing-optimizers/</guid>
- <description>Managing Optimizers The optimizer is the execution unit for performing self-optimizing tasks on a table. To isolate optimizing tasks on different tables and support the deployment of optimizers in different environments, Amoro has proposed the concepts of optimizer containers and optimizer groups:
Optimizer container: Encapsulate the deployment method of optimizers, there are four implementations for now: flink container based on Flink streaming job, spark containe [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Customized Encryption Method for Configurations</title>
<link>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</guid>
- <description>Using Customized Encryption Method for Configurations To enhance security, AMS allows encrypted sensitive configuration items such as passwords. Currently, AMS only supports the built-in base64 encryption algorithm (see Configure encrypted configuration items for details). If you require a stronger or customized encryption method, AMS also provides the flexibility to implement your own encryption algorithm.
Develop the Custom Implementation To integrate a custom en [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Kyuubi By Terminal</title>
<link>https://amoro.apache.org/docs/latest/using-kyuubi/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-kyuubi/</guid>
- <description>Using Kyuubi By Terminal Prerequisites:
There must be a running Kyuubi. To deploy and run Kyuubi, please refer to Kyuubi doc Terminal supports interfacing with Kyuubi to submit SQL to Kyuubi for execution. All you need to do is add the Kyuubi configuration as instructed below:
ams: terminal: backend: kyuubi kyuubi.jdbc.url: jdbc:hive2://127.0.0.1:10009/ # kyuubi Connection Address Kerberos Authentication Amoro terminal uses the Kerberos authentication informati [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/admin-guides/managing-catalogs/index.html b/output/docs/latest/admin-guides/managing-catalogs/index.html
index 7f1bf2425..752dce683 100644
--- a/output/docs/latest/admin-guides/managing-catalogs/index.html
+++ b/output/docs/latest/admin-guides/managing-catalogs/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/managing-catalogs/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/managing-catalogs/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/managing-catalogs/">
</head>
diff --git a/output/docs/latest/admin-guides/managing-optimizers/index.html b/output/docs/latest/admin-guides/managing-optimizers/index.html
index edcee7698..08d29c27a 100644
--- a/output/docs/latest/admin-guides/managing-optimizers/index.html
+++ b/output/docs/latest/admin-guides/managing-optimizers/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/managing-optimizers/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/managing-optimizers/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/managing-optimizers/">
</head>
diff --git a/output/docs/latest/admin-guides/using-customized-encryption-method/index.html b/output/docs/latest/admin-guides/using-customized-encryption-method/index.html
index a58cbcd0f..01b931f3c 100644
--- a/output/docs/latest/admin-guides/using-customized-encryption-method/index.html
+++ b/output/docs/latest/admin-guides/using-customized-encryption-method/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/using-customized-encryption-method/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/using-customized-encryption-method/">
</head>
diff --git a/output/docs/latest/admin-guides/using-kyuubi/index.html b/output/docs/latest/admin-guides/using-kyuubi/index.html
index 847a8600e..23422f495 100644
--- a/output/docs/latest/admin-guides/using-kyuubi/index.html
+++ b/output/docs/latest/admin-guides/using-kyuubi/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/using-kyuubi/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/using-kyuubi/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/using-kyuubi/">
</head>
diff --git a/output/docs/latest/ams-config/index.html b/output/docs/latest/ams-config/index.html
index 5a885363f..3e031d4d8 100644
--- a/output/docs/latest/ams-config/index.html
+++ b/output/docs/latest/ams-config/index.html
@@ -643,415 +643,415 @@ table td:nth-child(2), table th:nth-child(2) { width: 20%; word-break: break-all
table td:last-child, table th:last-child { width: 40%; word-break: break-all; }
</style>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>admin-password</td>
-<td>admin</td>
-<td>The administrator password</td>
-</tr>
-<tr>
-<td>admin-username</td>
-<td>admin</td>
-<td>The administrator account name.</td>
-</tr>
-<tr>
-<td>auto-create-tags.enabled</td>
-<td>true</td>
-<td>Enable creating tags.</td>
-</tr>
-<tr>
-<td>auto-create-tags.interval</td>
-<td>1 min</td>
-<td>Interval for creating tags.</td>
-</tr>
-<tr>
-<td>auto-create-tags.thread-count</td>
-<td>3</td>
-<td>The number of threads used for creating tags.</td>
-</tr>
-<tr>
-<td>blocker.timeout</td>
-<td>1 min</td>
-<td>Session timeout. Default unit is milliseconds if not specified.</td>
-</tr>
-<tr>
-<td>catalog-meta-cache.expiration-interval</td>
-<td>1 min</td>
-<td>TTL for catalog metadata.</td>
-</tr>
-<tr>
-<td>clean-dangling-delete-files.enabled</td>
-<td>true</td>
-<td>Enable dangling delete files cleaning.</td>
-</tr>
-<tr>
-<td>clean-dangling-delete-files.thread-count</td>
-<td>10</td>
-<td>The number of threads used for dangling delete files cleaning.</td>
-</tr>
-<tr>
-<td>clean-orphan-files.enabled</td>
-<td>true</td>
-<td>Enable orphan files cleaning.</td>
-</tr>
-<tr>
-<td>clean-orphan-files.interval</td>
-<td>1 d</td>
-<td>Interval for cleaning orphan files.</td>
-</tr>
-<tr>
-<td>clean-orphan-files.thread-count</td>
-<td>10</td>
-<td>The number of threads used for orphan files cleaning.</td>
-</tr>
-<tr>
-<td>data-expiration.enabled</td>
-<td>true</td>
-<td>Enable data expiration</td>
-</tr>
-<tr>
-<td>data-expiration.interval</td>
-<td>1 d</td>
-<td>Execute interval for data expiration</td>
-</tr>
-<tr>
-<td>data-expiration.thread-count</td>
-<td>10</td>
-<td>The number of threads used for data expiring</td>
-</tr>
-<tr>
-<td>database.auto-create-tables</td>
-<td>true</td>
-<td>Auto init table schema when started</td>
-</tr>
-<tr>
-<td>database.connection-pool-max-idle</td>
-<td>16</td>
-<td>Max idle connect count of database connect pool.</td>
-</tr>
-<tr>
-<td>database.connection-pool-max-total</td>
-<td>20</td>
-<td>Max connect count of database connect pool.</td>
-</tr>
-<tr>
-<td>database.connection-pool-max-wait-millis</td>
-<td>30000</td>
-<td>Max wait time before getting a connection timeout.</td>
-</tr>
-<tr>
-<td>database.jdbc-driver-class</td>
-<td>org.apache.derby.jdbc.EmbeddedDriver</td>
-<td>The JDBC driver class name for connecting to the database.</td>
-</tr>
-<tr>
-<td>database.password</td>
-<td></td>
-<td>The password for connecting to the database.</td>
-</tr>
-<tr>
-<td>database.type</td>
-<td>derby</td>
-<td>Database type.</td>
-</tr>
-<tr>
-<td>database.url</td>
-<td>jdbc:derby:/tmp/amoro/derby;create=true</td>
-<td>Database connection address</td>
-</tr>
-<tr>
-<td>database.username</td>
-<td>root</td>
-<td>The username for connecting to the database.</td>
-</tr>
-<tr>
-<td>expire-snapshots.enabled</td>
-<td>true</td>
-<td>Enable snapshots expiring.</td>
-</tr>
-<tr>
-<td>expire-snapshots.thread-count</td>
-<td>10</td>
-<td>The number of threads used for snapshots expiring.</td>
-</tr>
-<tr>
-<td>ha.cluster-name</td>
-<td>default</td>
-<td>Amoro management service cluster name.</td>
-</tr>
-<tr>
-<td>ha.connection-timeout</td>
-<td>5 min</td>
-<td>The Zookeeper connection timeout in milliseconds.</td>
-</tr>
-<tr>
-<td>ha.enabled</td>
-<td>false</td>
-<td>Whether to enable high availability mode.</td>
-</tr>
-<tr>
-<td>ha.heartbeat-interval</td>
-<td>10 s</td>
-<td>HA heartbeat interval.</td>
-</tr>
-<tr>
-<td>ha.lease-ttl</td>
-<td>30 s</td>
-<td>TTL of HA lease.</td>
-</tr>
-<tr>
-<td>ha.session-timeout</td>
-<td>30 s</td>
-<td>The Zookeeper session timeout in milliseconds.</td>
-</tr>
-<tr>
-<td>ha.type</td>
-<td>zk</td>
-<td>High availability implementation type: zk or database.</td>
-</tr>
-<tr>
-<td>ha.zookeeper-address</td>
-<td></td>
-<td>The Zookeeper address used for high availability.</td>
-</tr>
-<tr>
-<td>ha.zookeeper-auth-keytab</td>
-<td></td>
-<td>The Zookeeper authentication keytab file path when auth type is KERBEROS.</td>
-</tr>
-<tr>
-<td>ha.zookeeper-auth-principal</td>
-<td></td>
-<td>The Zookeeper authentication principal when auth type is KERBEROS.</td>
-</tr>
-<tr>
-<td>ha.zookeeper-auth-type</td>
-<td>NONE</td>
-<td>The Zookeeper authentication type, NONE or KERBEROS.</td>
-</tr>
-<tr>
-<td>http-server.auth-basic-provider</td>
-<td>org.apache.amoro.server.authentication.DefaultPasswdAuthenticationProvider</td>
-<td>User-defined password authentication implementation of org.apache.amoro.authentication.PasswdAuthenticationProvider</td>
-</tr>
-<tr>
-<td>http-server.auth-jwt-provider</td>
-<td><undefined></td>
-<td>User-defined JWT (JSON Web Token) authentication implementation of org.apache.amoro.authentication.TokenAuthenticationProvider</td>
-</tr>
-<tr>
-<td>http-server.bind-port</td>
-<td>19090</td>
-<td>Port that the Http server is bound to.</td>
-</tr>
-<tr>
-<td>http-server.proxy-client-ip-header</td>
-<td>X-Real-IP</td>
-<td>The HTTP header to record the real client IP address. If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP header that will be added to the request for other devices to use.</td>
-</tr>
-<tr>
-<td>http-server.rest-auth-type</td>
-<td>token</td>
-<td>The authentication used by REST APIs, token (default), basic or jwt.</td>
-</tr>
-<tr>
-<td>http-server.session-timeout</td>
-<td>7 d</td>
-<td>Timeout for http session.</td>
-</tr>
-<tr>
-<td>optimizer.heart-beat-timeout</td>
-<td>1 min</td>
-<td>Timeout duration for Optimizer heartbeat.</td>
-</tr>
-<tr>
-<td>optimizer.max-planning-parallelism</td>
-<td>1</td>
-<td>Max planning parallelism in one optimizer group.</td>
-</tr>
-<tr>
-<td>optimizer.polling-timeout</td>
-<td>3 s</td>
-<td>Optimizer polling task timeout.</td>
-</tr>
-<tr>
-<td>optimizer.task-ack-timeout</td>
-<td>30 s</td>
-<td>Timeout duration for task acknowledgment.</td>
-</tr>
-<tr>
-<td>optimizer.task-execute-timeout</td>
-<td>1 h</td>
-<td>Timeout duration for task execution, default to 1 hour.</td>
-</tr>
-<tr>
-<td>overview-cache.max-size</td>
-<td>3360</td>
-<td>Max size of overview cache.</td>
-</tr>
-<tr>
-<td>overview-cache.refresh-interval</td>
-<td>3 min</td>
-<td>Interval for refreshing overview cache.</td>
-</tr>
-<tr>
-<td>refresh-external-catalogs.interval</td>
-<td>3 min</td>
-<td>Interval to refresh the external catalog.</td>
-</tr>
-<tr>
-<td>refresh-external-catalogs.queue-size</td>
-<td>1000000</td>
-<td>The queue size of the executors of the external catalog explorer.</td>
-</tr>
-<tr>
-<td>refresh-external-catalogs.thread-count</td>
-<td>10</td>
-<td>The number of threads used for discovering tables in external catalogs.</td>
-</tr>
-<tr>
-<td>refresh-tables.interval</td>
-<td>1 min</td>
-<td>Interval for refreshing table metadata.</td>
-</tr>
-<tr>
-<td>refresh-tables.max-pending-partition-count</td>
-<td>100</td>
-<td>Filters will not be used beyond that number of partitions.</td>
-</tr>
-<tr>
-<td>refresh-tables.thread-count</td>
-<td>10</td>
-<td>The number of threads used for refreshing tables.</td>
-</tr>
-<tr>
-<td>self-optimizing.break-quota-limit-enabled</td>
-<td>true</td>
-<td>Allow the table to break the quota limit when the resource is sufficient.</td>
-</tr>
-<tr>
-<td>self-optimizing.commit-manifest-io-thread-count</td>
-<td>10</td>
-<td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent commit operations.</td>
-</tr>
-<tr>
-<td>self-optimizing.commit-thread-count</td>
-<td>10</td>
-<td>The number of threads that self-optimizing uses to submit results.</td>
-</tr>
-<tr>
-<td>self-optimizing.plan-manifest-io-thread-count</td>
-<td>10</td>
-<td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent planning operations.</td>
-</tr>
-<tr>
-<td>self-optimizing.refresh-group-interval</td>
-<td>30 s</td>
-<td>Optimizer group refresh interval.</td>
-</tr>
-<tr>
-<td>self-optimizing.runtime-data-expire-interval-hours</td>
-<td>1</td>
-<td>The number of hours that self-optimizing runtime data expire interval.</td>
-</tr>
-<tr>
-<td>self-optimizing.runtime-data-keep-days</td>
-<td>30</td>
-<td>The number of days that self-optimizing runtime data keeps the runtime.</td>
-</tr>
-<tr>
-<td>server-bind-host</td>
-<td>0.0.0.0</td>
-<td>The host bound to the server.</td>
-</tr>
-<tr>
-<td>server-expose-host</td>
-<td></td>
-<td>The exposed host of the server.</td>
-</tr>
-<tr>
-<td>sync-hive-tables.enabled</td>
-<td>false</td>
-<td>Enable synchronizing Hive tables.</td>
-</tr>
-<tr>
-<td>sync-hive-tables.thread-count</td>
-<td>10</td>
-<td>The number of threads used for synchronizing Hive tables.</td>
-</tr>
-<tr>
-<td>table-manifest-io.thread-count</td>
-<td>20</td>
-<td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent planning or commit operations.</td>
-</tr>
-<tr>
-<td>terminal.backend</td>
-<td>local</td>
-<td>Terminal backend implementation. local, kyuubi and custom are valid values.</td>
-</tr>
-<tr>
-<td>terminal.factory</td>
-<td><undefined></td>
-<td>Session factory implement of terminal, <code>terminal.backend</code> must be <code>custom</code> if this is set.</td>
-</tr>
-<tr>
-<td>terminal.result.limit</td>
-<td>1000</td>
-<td>Row limit of result-set</td>
-</tr>
-<tr>
-<td>terminal.sensitive-conf-keys</td>
-<td></td>
-<td>Comma-separated list of sensitive conf keys used to desensitize related value.</td>
-</tr>
-<tr>
-<td>terminal.session.timeout</td>
-<td>30 min</td>
-<td>Session timeout. Default unit is milliseconds if not specified (** Note: default units are minutes when version < 0.8).</td>
-</tr>
-<tr>
-<td>terminal.stop-on-error</td>
-<td>false</td>
-<td>When a statement fails to execute, stop execution or continue executing the remaining statements.</td>
-</tr>
-<tr>
-<td>thrift-server.max-message-size</td>
-<td>100 mb</td>
-<td>Maximum message size that the Thrift server can accept. Default unit is bytes if not specified.</td>
-</tr>
-<tr>
-<td>thrift-server.optimizing-service.bind-port</td>
-<td>1261</td>
-<td>Port that the optimizing service thrift server is bound to.</td>
-</tr>
-<tr>
-<td>thrift-server.selector-queue-size</td>
-<td>4</td>
-<td>The number of queue size per selector thread for the Thrift server</td>
-</tr>
-<tr>
-<td>thrift-server.selector-thread-count</td>
-<td>2</td>
-<td>The number of selector threads for the Thrift server.</td>
-</tr>
-<tr>
-<td>thrift-server.table-service.bind-port</td>
-<td>1260</td>
-<td>Port that the table service thrift server is bound to.</td>
-</tr>
-<tr>
-<td>thrift-server.table-service.worker-thread-count</td>
-<td>20</td>
-<td>The number of worker threads for the Thrift server.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>admin-password</td>
+ <td>admin</td>
+ <td>The administrator password</td>
+ </tr>
+ <tr>
+ <td>admin-username</td>
+ <td>admin</td>
+ <td>The administrator account name.</td>
+ </tr>
+ <tr>
+ <td>auto-create-tags.enabled</td>
+ <td>true</td>
+ <td>Enable creating tags.</td>
+ </tr>
+ <tr>
+ <td>auto-create-tags.interval</td>
+ <td>1 min</td>
+ <td>Interval for creating tags.</td>
+ </tr>
+ <tr>
+ <td>auto-create-tags.thread-count</td>
+ <td>3</td>
+ <td>The number of threads used for creating tags.</td>
+ </tr>
+ <tr>
+ <td>blocker.timeout</td>
+ <td>1 min</td>
+ <td>Session timeout. Default unit is milliseconds if not specified.</td>
+ </tr>
+ <tr>
+ <td>catalog-meta-cache.expiration-interval</td>
+ <td>1 min</td>
+ <td>TTL for catalog metadata.</td>
+ </tr>
+ <tr>
+ <td>clean-dangling-delete-files.enabled</td>
+ <td>true</td>
+ <td>Enable dangling delete files cleaning.</td>
+ </tr>
+ <tr>
+ <td>clean-dangling-delete-files.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for dangling delete files cleaning.</td>
+ </tr>
+ <tr>
+ <td>clean-orphan-files.enabled</td>
+ <td>true</td>
+ <td>Enable orphan files cleaning.</td>
+ </tr>
+ <tr>
+ <td>clean-orphan-files.interval</td>
+ <td>1 d</td>
+ <td>Interval for cleaning orphan files.</td>
+ </tr>
+ <tr>
+ <td>clean-orphan-files.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for orphan files cleaning.</td>
+ </tr>
+ <tr>
+ <td>data-expiration.enabled</td>
+ <td>true</td>
+ <td>Enable data expiration</td>
+ </tr>
+ <tr>
+ <td>data-expiration.interval</td>
+ <td>1 d</td>
+ <td>Execute interval for data expiration</td>
+ </tr>
+ <tr>
+ <td>data-expiration.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for data expiring</td>
+ </tr>
+ <tr>
+ <td>database.auto-create-tables</td>
+ <td>true</td>
+ <td>Auto init table schema when started</td>
+ </tr>
+ <tr>
+ <td>database.connection-pool-max-idle</td>
+ <td>16</td>
+ <td>Max idle connect count of database connect pool.</td>
+ </tr>
+ <tr>
+ <td>database.connection-pool-max-total</td>
+ <td>20</td>
+ <td>Max connect count of database connect pool.</td>
+ </tr>
+ <tr>
+ <td>database.connection-pool-max-wait-millis</td>
+ <td>30000</td>
+ <td>Max wait time before getting a connection timeout.</td>
+ </tr>
+ <tr>
+ <td>database.jdbc-driver-class</td>
+ <td>org.apache.derby.jdbc.EmbeddedDriver</td>
+ <td>The JDBC driver class name for connecting to the database.</td>
+ </tr>
+ <tr>
+ <td>database.password</td>
+ <td></td>
+ <td>The password for connecting to the database.</td>
+ </tr>
+ <tr>
+ <td>database.type</td>
+ <td>derby</td>
+ <td>Database type.</td>
+ </tr>
+ <tr>
+ <td>database.url</td>
+ <td>jdbc:derby:/tmp/amoro/derby;create=true</td>
+ <td>Database connection address</td>
+ </tr>
+ <tr>
+ <td>database.username</td>
+ <td>root</td>
+ <td>The username for connecting to the database.</td>
+ </tr>
+ <tr>
+ <td>expire-snapshots.enabled</td>
+ <td>true</td>
+ <td>Enable snapshots expiring.</td>
+ </tr>
+ <tr>
+ <td>expire-snapshots.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for snapshots expiring.</td>
+ </tr>
+ <tr>
+ <td>ha.cluster-name</td>
+ <td>default</td>
+ <td>Amoro management service cluster name.</td>
+ </tr>
+ <tr>
+ <td>ha.connection-timeout</td>
+ <td>5 min</td>
+ <td>The Zookeeper connection timeout in milliseconds.</td>
+ </tr>
+ <tr>
+ <td>ha.enabled</td>
+ <td>false</td>
+ <td>Whether to enable high availability mode.</td>
+ </tr>
+ <tr>
+ <td>ha.heartbeat-interval</td>
+ <td>10 s</td>
+ <td>HA heartbeat interval.</td>
+ </tr>
+ <tr>
+ <td>ha.lease-ttl</td>
+ <td>30 s</td>
+ <td>TTL of HA lease.</td>
+ </tr>
+ <tr>
+ <td>ha.session-timeout</td>
+ <td>30 s</td>
+ <td>The Zookeeper session timeout in milliseconds.</td>
+ </tr>
+ <tr>
+ <td>ha.type</td>
+ <td>zk</td>
+ <td>High availability implementation type: zk or database.</td>
+ </tr>
+ <tr>
+ <td>ha.zookeeper-address</td>
+ <td></td>
+ <td>The Zookeeper address used for high availability.</td>
+ </tr>
+ <tr>
+ <td>ha.zookeeper-auth-keytab</td>
+ <td></td>
+ <td>The Zookeeper authentication keytab file path when auth type is KERBEROS.</td>
+ </tr>
+ <tr>
+ <td>ha.zookeeper-auth-principal</td>
+ <td></td>
+ <td>The Zookeeper authentication principal when auth type is KERBEROS.</td>
+ </tr>
+ <tr>
+ <td>ha.zookeeper-auth-type</td>
+ <td>NONE</td>
+ <td>The Zookeeper authentication type, NONE or KERBEROS.</td>
+ </tr>
+ <tr>
+ <td>http-server.auth-basic-provider</td>
+ <td>org.apache.amoro.server.authentication.DefaultPasswdAuthenticationProvider</td>
+ <td>User-defined password authentication implementation of org.apache.amoro.authentication.PasswdAuthenticationProvider</td>
+ </tr>
+ <tr>
+ <td>http-server.auth-jwt-provider</td>
+ <td><undefined></td>
+ <td>User-defined JWT (JSON Web Token) authentication implementation of org.apache.amoro.authentication.TokenAuthenticationProvider</td>
+ </tr>
+ <tr>
+ <td>http-server.bind-port</td>
+ <td>19090</td>
+ <td>Port that the Http server is bound to.</td>
+ </tr>
+ <tr>
+ <td>http-server.proxy-client-ip-header</td>
+ <td>X-Real-IP</td>
+ <td>The HTTP header to record the real client IP address. If your server is behind a load balancer or other proxy, the server will see this load balancer or proxy IP address as the client IP address, to get around this common issue, most load balancers or proxies offer the ability to record the real remote IP address in an HTTP header that will be added to the request for other devices to use.</td>
+ </tr>
+ <tr>
+ <td>http-server.rest-auth-type</td>
+ <td>token</td>
+ <td>The authentication used by REST APIs, token (default), basic or jwt.</td>
+ </tr>
+ <tr>
+ <td>http-server.session-timeout</td>
+ <td>7 d</td>
+ <td>Timeout for http session.</td>
+ </tr>
+ <tr>
+ <td>optimizer.heart-beat-timeout</td>
+ <td>1 min</td>
+ <td>Timeout duration for Optimizer heartbeat.</td>
+ </tr>
+ <tr>
+ <td>optimizer.max-planning-parallelism</td>
+ <td>1</td>
+ <td>Max planning parallelism in one optimizer group.</td>
+ </tr>
+ <tr>
+ <td>optimizer.polling-timeout</td>
+ <td>3 s</td>
+ <td>Optimizer polling task timeout.</td>
+ </tr>
+ <tr>
+ <td>optimizer.task-ack-timeout</td>
+ <td>30 s</td>
+ <td>Timeout duration for task acknowledgment.</td>
+ </tr>
+ <tr>
+ <td>optimizer.task-execute-timeout</td>
+ <td>1 h</td>
+ <td>Timeout duration for task execution, default to 1 hour.</td>
+ </tr>
+ <tr>
+ <td>overview-cache.max-size</td>
+ <td>3360</td>
+ <td>Max size of overview cache.</td>
+ </tr>
+ <tr>
+ <td>overview-cache.refresh-interval</td>
+ <td>3 min</td>
+ <td>Interval for refreshing overview cache.</td>
+ </tr>
+ <tr>
+ <td>refresh-external-catalogs.interval</td>
+ <td>3 min</td>
+ <td>Interval to refresh the external catalog.</td>
+ </tr>
+ <tr>
+ <td>refresh-external-catalogs.queue-size</td>
+ <td>1000000</td>
+ <td>The queue size of the executors of the external catalog explorer.</td>
+ </tr>
+ <tr>
+ <td>refresh-external-catalogs.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for discovering tables in external catalogs.</td>
+ </tr>
+ <tr>
+ <td>refresh-tables.interval</td>
+ <td>1 min</td>
+ <td>Interval for refreshing table metadata.</td>
+ </tr>
+ <tr>
+ <td>refresh-tables.max-pending-partition-count</td>
+ <td>100</td>
+ <td>Filters will not be used beyond that number of partitions.</td>
+ </tr>
+ <tr>
+ <td>refresh-tables.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for refreshing tables.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.break-quota-limit-enabled</td>
+ <td>true</td>
+ <td>Allow the table to break the quota limit when the resource is sufficient.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.commit-manifest-io-thread-count</td>
+ <td>10</td>
+ <td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent commit operations.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.commit-thread-count</td>
+ <td>10</td>
+ <td>The number of threads that self-optimizing uses to submit results.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.plan-manifest-io-thread-count</td>
+ <td>10</td>
+ <td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent planning operations.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.refresh-group-interval</td>
+ <td>30 s</td>
+ <td>Optimizer group refresh interval.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.runtime-data-expire-interval-hours</td>
+ <td>1</td>
+ <td>The number of hours that self-optimizing runtime data expire interval.</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.runtime-data-keep-days</td>
+ <td>30</td>
+ <td>The number of days that self-optimizing runtime data keeps the runtime.</td>
+ </tr>
+ <tr>
+ <td>server-bind-host</td>
+ <td>0.0.0.0</td>
+ <td>The host bound to the server.</td>
+ </tr>
+ <tr>
+ <td>server-expose-host</td>
+ <td></td>
+ <td>The exposed host of the server.</td>
+ </tr>
+ <tr>
+ <td>sync-hive-tables.enabled</td>
+ <td>false</td>
+ <td>Enable synchronizing Hive tables.</td>
+ </tr>
+ <tr>
+ <td>sync-hive-tables.thread-count</td>
+ <td>10</td>
+ <td>The number of threads used for synchronizing Hive tables.</td>
+ </tr>
+ <tr>
+ <td>table-manifest-io.thread-count</td>
+ <td>20</td>
+ <td>Sets the size of the worker pool. The worker pool limits the number of tasks concurrently processing manifests in the base table implementation across all concurrent planning or commit operations.</td>
+ </tr>
+ <tr>
+ <td>terminal.backend</td>
+ <td>local</td>
+ <td>Terminal backend implementation. local, kyuubi and custom are valid values.</td>
+ </tr>
+ <tr>
+ <td>terminal.factory</td>
+ <td><undefined></td>
+ <td>Session factory implement of terminal, <code>terminal.backend</code> must be <code>custom</code> if this is set.</td>
+ </tr>
+ <tr>
+ <td>terminal.result.limit</td>
+ <td>1000</td>
+ <td>Row limit of result-set</td>
+ </tr>
+ <tr>
+ <td>terminal.sensitive-conf-keys</td>
+ <td></td>
+ <td>Comma-separated list of sensitive conf keys used to desensitize related value.</td>
+ </tr>
+ <tr>
+ <td>terminal.session.timeout</td>
+ <td>30 min</td>
+ <td>Session timeout. Default unit is milliseconds if not specified (** Note: default units are minutes when version < 0.8).</td>
+ </tr>
+ <tr>
+ <td>terminal.stop-on-error</td>
+ <td>false</td>
+ <td>When a statement fails to execute, stop execution or continue executing the remaining statements.</td>
+ </tr>
+ <tr>
+ <td>thrift-server.max-message-size</td>
+ <td>100 mb</td>
+ <td>Maximum message size that the Thrift server can accept. Default unit is bytes if not specified.</td>
+ </tr>
+ <tr>
+ <td>thrift-server.optimizing-service.bind-port</td>
+ <td>1261</td>
+ <td>Port that the optimizing service thrift server is bound to.</td>
+ </tr>
+ <tr>
+ <td>thrift-server.selector-queue-size</td>
+ <td>4</td>
+ <td>The number of queue size per selector thread for the Thrift server</td>
+ </tr>
+ <tr>
+ <td>thrift-server.selector-thread-count</td>
+ <td>2</td>
+ <td>The number of selector threads for the Thrift server.</td>
+ </tr>
+ <tr>
+ <td>thrift-server.table-service.bind-port</td>
+ <td>1260</td>
+ <td>Port that the table service thrift server is bound to.</td>
+ </tr>
+ <tr>
+ <td>thrift-server.table-service.worker-thread-count</td>
+ <td>20</td>
+ <td>The number of worker threads for the Thrift server.</td>
+ </tr>
+ </tbody>
</table>
<h2 id="shade-utils-configuration">Shade Utils Configuration</h2>
<p>The configuration options for Amoro Configuration Shade Utils.</p>
@@ -1063,25 +1063,25 @@ table td:nth-child(2), table th:nth-child(2) { width: 20%; word-break: break-all
table td:last-child, table th:last-child { width: 40%; word-break: break-all; }
</style>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>shade.identifier</td>
-<td>default</td>
-<td>The identifier of the encryption method for decryption. Defaults to “default”, indicating no encryption</td>
-</tr>
-<tr>
-<td>shade.sensitive-keywords</td>
-<td>admin-password;database.password</td>
-<td>A semicolon-separated list of keywords for the configuration items to be decrypted.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>shade.identifier</td>
+ <td>default</td>
+ <td>The identifier of the encryption method for decryption. Defaults to “default”, indicating no encryption</td>
+ </tr>
+ <tr>
+ <td>shade.sensitive-keywords</td>
+ <td>admin-password;database.password</td>
+ <td>A semicolon-separated list of keywords for the configuration items to be decrypted.</td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/concept/catalogs/index.html b/output/docs/latest/concept/catalogs/index.html
index 8f6d8ea77..6c76ea96f 100644
--- a/output/docs/latest/concept/catalogs/index.html
+++ b/output/docs/latest/concept/catalogs/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/catalogs/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/catalogs/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/catalogs/">
</head>
diff --git a/output/docs/latest/concept/self-optimizing/index.html b/output/docs/latest/concept/self-optimizing/index.html
index 45cf651ee..f395e080f 100644
--- a/output/docs/latest/concept/self-optimizing/index.html
+++ b/output/docs/latest/concept/self-optimizing/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/self-optimizing/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/self-optimizing/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/self-optimizing/">
</head>
diff --git a/output/docs/latest/concept/table-watermark/index.html b/output/docs/latest/concept/table-watermark/index.html
index bd6255ffa..e3651e206 100644
--- a/output/docs/latest/concept/table-watermark/index.html
+++ b/output/docs/latest/concept/table-watermark/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/table-watermark/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/table-watermark/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/table-watermark/">
</head>
diff --git a/output/docs/latest/concepts/index.xml b/output/docs/latest/concepts/index.xml
index 14afc4309..ffec10bc6 100644
--- a/output/docs/latest/concepts/index.xml
+++ b/output/docs/latest/concepts/index.xml
@@ -12,21 +12,21 @@
<link>https://amoro.apache.org/docs/latest/catalogs/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/catalogs/</guid>
- <description>Catalogs Introduce multi-catalog A catalog is a metadata namespace that stores information about databases, tables, views, indexes, users, and UDFs. It provides a higher-level namespace for table and database. Typically, a catalog is associated with a specific type of data source or cluster. In Flink, Spark and Trino, the multi-catalog feature can be used to support SQL across data sources, such as:
SELECT c.ID, c.NAME, c.AGE, o.AMOUNT FROM ${CATALOG_A}.ONLINE.CUST [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Self-Optimizing</title>
<link>https://amoro.apache.org/docs/latest/self-optimizing/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/self-optimizing/</guid>
- <description>Self-optimizing Introduction Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time analysis scenarios, challenges arise. For example:
Streaming writes bring a massive amount of fragment files CDC ingestion and streaming updates generat [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Table Watermark</title>
<link>https://amoro.apache.org/docs/latest/table-watermark/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/table-watermark/</guid>
- <description>Table Watermark Table freshness Data freshness represents timeliness, and in many discussions, freshness is considered one of the important indicators of data quality. In traditional offline data warehouses, higher cost typically means better performance, creating a typical binary paradox in terms of cost-performance trade-off. However, in high-freshness streaming data warehouses, massive small files and frequent updates can lead to performance degradation. The higher [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/configuration/ams-config/index.html b/output/docs/latest/configuration/ams-config/index.html
index 1106bf477..ec8c19a4f 100644
--- a/output/docs/latest/configuration/ams-config/index.html
+++ b/output/docs/latest/configuration/ams-config/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/ams-config/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/ams-config/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/ams-config/">
</head>
diff --git a/output/docs/latest/configuration/index.xml b/output/docs/latest/configuration/index.xml
index 33ca5c61b..d55ac800e 100644
--- a/output/docs/latest/configuration/index.xml
+++ b/output/docs/latest/configuration/index.xml
@@ -12,7 +12,7 @@
<link>https://amoro.apache.org/docs/latest/ams-config/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/ams-config/</guid>
- <description>AMS Configuration Amoro Management Service Configuration The configuration options for Amoro Management Service (AMS).
Key Default Description admin-password admin The administrator password admin-username admin The administrator account name. auto-create-tags.enabled true Enable creating tags. auto-create-tags.interval 1 min Interval for creating tags. auto-create-tags.thread-count 3 The number of threads used for creating tags. blocker.timeout 1 min Session timeo [...]
+ <description><!--
 ~ Licensed to the Apache Software Foundation (ASF) under one
 ~ or more contributor license agreements. See the NOTICE file
 ~ distributed with this work for additional information
 ~ regarding copyright ownership. The ASF licenses this file
 ~ to you under the Apache License, Version 2.0 (the
 ~ "License"); you may not use this file except in compliance
 ~ with the License. You may obtain a copy of the License at& [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/configurations/index.html b/output/docs/latest/configurations/index.html
index 4bcf20f44..fe7821409 100644
--- a/output/docs/latest/configurations/index.html
+++ b/output/docs/latest/configurations/index.html
@@ -646,429 +646,429 @@ modified through <a href="../using-tables/#modify-table">Alter Table</a> operati
<h2 id="self-optimizing-configurations">Self-optimizing configurations</h2>
<p>Self-optimizing configurations are applicable to both Iceberg Format and Mixed streaming Format.</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>self-optimizing.enabled</td>
-<td>true</td>
-<td>Enables Self-optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.allow-partial-commit</td>
-<td>false</td>
-<td>Whether to allow partial commit when self-optimizing fails or process is cancelled</td>
-</tr>
-<tr>
-<td>self-optimizing.group</td>
-<td>default</td>
-<td>Optimizer group for Self-optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.quota</td>
-<td>0.5</td>
-<td>Quota for Self-optimizing, indicating the optimizer resources the table can take up</td>
-</tr>
-<tr>
-<td>self-optimizing.execute.num-retries</td>
-<td>5</td>
-<td>Number of retries after failure of Self-optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.target-size</td>
-<td>134217728(128MB)</td>
-<td>Target size for Self-optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.max-file-count</td>
-<td>10000</td>
-<td>Maximum number of files processed by a Self-optimizing process</td>
-</tr>
-<tr>
-<td>self-optimizing.max-task-size-bytes</td>
-<td>134217728(128MB)</td>
-<td>Maximum file size bytes in a single task for splitting tasks</td>
-</tr>
-<tr>
-<td>self-optimizing.fragment-ratio</td>
-<td>8</td>
-<td>The fragment file size threshold. We could divide self-optimizing.target-size by this ratio to get the actual fragment file size</td>
-</tr>
-<tr>
-<td>self-optimizing.min-target-size-ratio</td>
-<td>0.75</td>
-<td>The undersized segment file size threshold. Segment files under this threshold will be considered for rewriting</td>
-</tr>
-<tr>
-<td>self-optimizing.minor.trigger.file-count</td>
-<td>12</td>
-<td>The minimum number of files to trigger minor optimizing is determined by the sum of fragment file count and equality delete file count</td>
-</tr>
-<tr>
-<td>self-optimizing.minor.trigger.interval</td>
-<td>3600000(1 hour)</td>
-<td>The time interval in milliseconds to trigger minor optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.major.trigger.duplicate-ratio</td>
-<td>0.1</td>
-<td>The ratio of duplicate data of segment files to trigger major optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.full.trigger.interval</td>
-<td>-1(closed)</td>
-<td>The time interval in milliseconds to trigger full optimizing</td>
-</tr>
-<tr>
-<td>self-optimizing.full.rewrite-all-files</td>
-<td>true</td>
-<td>Whether full optimizing rewrites all files or skips files that do not need to be optimized</td>
-</tr>
-<tr>
-<td>self-optimizing.min-plan-interval</td>
-<td>60000</td>
-<td>The minimum time interval between two self-optimizing planning action</td>
-</tr>
-<tr>
-<td>self-optimizing.filter</td>
-<td>NULL</td>
-<td>Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > ‘2007-12-03T10:15:30’.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>self-optimizing.enabled</td>
+ <td>true</td>
+ <td>Enables Self-optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.allow-partial-commit</td>
+ <td>false</td>
+ <td>Whether to allow partial commit when self-optimizing fails or process is cancelled</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.group</td>
+ <td>default</td>
+ <td>Optimizer group for Self-optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.quota</td>
+ <td>0.5</td>
+ <td>Quota for Self-optimizing, indicating the optimizer resources the table can take up</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.execute.num-retries</td>
+ <td>5</td>
+ <td>Number of retries after failure of Self-optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.target-size</td>
+ <td>134217728(128MB)</td>
+ <td>Target size for Self-optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.max-file-count</td>
+ <td>10000</td>
+ <td>Maximum number of files processed by a Self-optimizing process</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.max-task-size-bytes</td>
+ <td>134217728(128MB)</td>
+ <td>Maximum file size bytes in a single task for splitting tasks</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.fragment-ratio</td>
+ <td>8</td>
+ <td>The fragment file size threshold. We could divide self-optimizing.target-size by this ratio to get the actual fragment file size</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.min-target-size-ratio</td>
+ <td>0.75</td>
+ <td>The undersized segment file size threshold. Segment files under this threshold will be considered for rewriting</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.minor.trigger.file-count</td>
+ <td>12</td>
+ <td>The minimum number of files to trigger minor optimizing is determined by the sum of fragment file count and equality delete file count</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.minor.trigger.interval</td>
+ <td>3600000(1 hour)</td>
+ <td>The time interval in milliseconds to trigger minor optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.major.trigger.duplicate-ratio</td>
+ <td>0.1</td>
+ <td>The ratio of duplicate data of segment files to trigger major optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.full.trigger.interval</td>
+ <td>-1(closed)</td>
+ <td>The time interval in milliseconds to trigger full optimizing</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.full.rewrite-all-files</td>
+ <td>true</td>
+ <td>Whether full optimizing rewrites all files or skips files that do not need to be optimized</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.min-plan-interval</td>
+ <td>60000</td>
+ <td>The minimum time interval between two self-optimizing planning action</td>
+ </tr>
+ <tr>
+ <td>self-optimizing.filter</td>
+ <td>NULL</td>
+ <td>Filter conditions for self-optimizing, using SQL conditional expressions, without supporting any functions. For the timestamp column condition, the ISO date-time formatter must be used. For example: op_time > ‘2007-12-03T10:15:30’.</td>
+ </tr>
+ </tbody>
</table>
<h2 id="data-cleaning-configurations">Data-cleaning configurations</h2>
<p>Data-cleaning configurations are applicable to both Iceberg Format and Mixed streaming Format.</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>table-expire.enabled</td>
-<td>true</td>
-<td>Enables periodically expire table</td>
-</tr>
-<tr>
-<td>change.data.ttl.minutes</td>
-<td>10080(7 days)</td>
-<td>Time to live in minutes for data of ChangeStore</td>
-</tr>
-<tr>
-<td>snapshot.keep.duration</td>
-<td>720min(12 hours)</td>
-<td>Table-Expiration keeps the latest snapshots within a specified duration</td>
-</tr>
-<tr>
-<td>snapshot.keep.min-count</td>
-<td>1</td>
-<td>Minimum number of snapshots retained for table expiration</td>
-</tr>
-<tr>
-<td>snapshot.keep.flink.checkpoint-retention</td>
-<td>7d(7 days)</td>
-<td>The retention period for snapshots created by Flink checkpoints. Snapshots older than this duration may be cleaned up. The value should be specified as a duration string (e.g., “7d”, “168h”, “10080min”)</td>
-</tr>
-<tr>
-<td>clean-orphan-file.enabled</td>
-<td>false</td>
-<td>Enables periodically clean orphan files</td>
-</tr>
-<tr>
-<td>clean-orphan-file.min-existing-time-minutes</td>
-<td>2880(2 days)</td>
-<td>Cleaning orphan files keeps the files modified within a specified time in minutes</td>
-</tr>
-<tr>
-<td>clean-dangling-delete-files.enabled</td>
-<td>true</td>
-<td>Whether to enable cleaning of dangling delete files</td>
-</tr>
-<tr>
-<td>data-expire.enabled</td>
-<td>false</td>
-<td>Whether to enable data expiration</td>
-</tr>
-<tr>
-<td>data-expire.level</td>
-<td>partition</td>
-<td>Level of data expiration. Including partition and file</td>
-</tr>
-<tr>
-<td>data-expire.field</td>
-<td>NULL</td>
-<td>Field used to determine data expiration, supporting timestamp/timestampz/long type and string type field in date format</td>
-</tr>
-<tr>
-<td>data-expire.datetime-string-pattern</td>
-<td>yyyy-MM-dd</td>
-<td>Pattern used for matching string datetime</td>
-</tr>
-<tr>
-<td>data-expire.datetime-number-format</td>
-<td>TIMESTAMP_MS</td>
-<td>Timestamp unit for long field. Including TIMESTAMP_MS and TIMESTAMP_S</td>
-</tr>
-<tr>
-<td>data-expire.retention-time</td>
-<td>NULL</td>
-<td>Retention period for data expiration. For example, 1d means retaining data for 1 day. Other supported units include h (hour), min (minute), s (second), ms (millisecond), etc.</td>
-</tr>
-<tr>
-<td>data-expire.base-on-rule</td>
-<td>LAST_COMMIT_TIME</td>
-<td>A rule to indicate how to start expire data. Including LAST_COMMIT_TIME and CURRENT_TIME. LAST_COMMIT_TIME uses the timestamp of latest commit snapshot which is not optimized as the start of the expiration, which ensures that the table has <code>retention-time</code> data</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>table-expire.enabled</td>
+ <td>true</td>
+ <td>Enables periodically expire table</td>
+ </tr>
+ <tr>
+ <td>change.data.ttl.minutes</td>
+ <td>10080(7 days)</td>
+ <td>Time to live in minutes for data of ChangeStore</td>
+ </tr>
+ <tr>
+ <td>snapshot.keep.duration</td>
+ <td>720min(12 hours)</td>
+ <td>Table-Expiration keeps the latest snapshots within a specified duration</td>
+ </tr>
+ <tr>
+ <td>snapshot.keep.min-count</td>
+ <td>1</td>
+ <td>Minimum number of snapshots retained for table expiration</td>
+ </tr>
+ <tr>
+ <td>snapshot.keep.flink.checkpoint-retention</td>
+ <td>7d(7 days)</td>
+ <td>The retention period for snapshots created by Flink checkpoints. Snapshots older than this duration may be cleaned up. The value should be specified as a duration string (e.g., “7d”, “168h”, “10080min”)</td>
+ </tr>
+ <tr>
+ <td>clean-orphan-file.enabled</td>
+ <td>false</td>
+ <td>Enables periodically clean orphan files</td>
+ </tr>
+ <tr>
+ <td>clean-orphan-file.min-existing-time-minutes</td>
+ <td>2880(2 days)</td>
+ <td>Cleaning orphan files keeps the files modified within a specified time in minutes</td>
+ </tr>
+ <tr>
+ <td>clean-dangling-delete-files.enabled</td>
+ <td>true</td>
+ <td>Whether to enable cleaning of dangling delete files</td>
+ </tr>
+ <tr>
+ <td>data-expire.enabled</td>
+ <td>false</td>
+ <td>Whether to enable data expiration</td>
+ </tr>
+ <tr>
+ <td>data-expire.level</td>
+ <td>partition</td>
+ <td>Level of data expiration. Including partition and file</td>
+ </tr>
+ <tr>
+ <td>data-expire.field</td>
+ <td>NULL</td>
+ <td>Field used to determine data expiration, supporting timestamp/timestampz/long type and string type field in date format</td>
+ </tr>
+ <tr>
+ <td>data-expire.datetime-string-pattern</td>
+ <td>yyyy-MM-dd</td>
+ <td>Pattern used for matching string datetime</td>
+ </tr>
+ <tr>
+ <td>data-expire.datetime-number-format</td>
+ <td>TIMESTAMP_MS</td>
+ <td>Timestamp unit for long field. Including TIMESTAMP_MS and TIMESTAMP_S</td>
+ </tr>
+ <tr>
+ <td>data-expire.retention-time</td>
+ <td>NULL</td>
+ <td>Retention period for data expiration. For example, 1d means retaining data for 1 day. Other supported units include h (hour), min (minute), s (second), ms (millisecond), etc.</td>
+ </tr>
+ <tr>
+ <td>data-expire.base-on-rule</td>
+ <td>LAST_COMMIT_TIME</td>
+ <td>A rule to indicate how to start expire data. Including LAST_COMMIT_TIME and CURRENT_TIME. LAST_COMMIT_TIME uses the timestamp of latest commit snapshot which is not optimized as the start of the expiration, which ensures that the table has <code>retention-time</code> data</td>
+ </tr>
+ </tbody>
</table>
<h2 id="tags-configurations">Tags configurations</h2>
<p>Tags configurations are applicable to Iceberg Format only now, and will be supported in Mixed Format
soon.</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>tag.auto-create.enabled</td>
-<td>false</td>
-<td>Enables automatically creating tags</td>
-</tr>
-<tr>
-<td>tag.auto-create.trigger.period</td>
-<td>daily</td>
-<td>Period of creating tags, support <code>daily</code>,<code>hourly</code> now</td>
-</tr>
-<tr>
-<td>tag.auto-create.trigger.offset.minutes</td>
-<td>0</td>
-<td>The minutes by which the tag is created after midnight (00:00)</td>
-</tr>
-<tr>
-<td>tag.auto-create.trigger.max-delay.minutes</td>
-<td>60</td>
-<td>The maximum delay time for creating a tag</td>
-</tr>
-<tr>
-<td>tag.auto-create.tag-format</td>
-<td>’tag-‘yyyyMMdd for daily and ’tag-‘yyyyMMddHH for hourly periods</td>
-<td>The format of the name for tag. Modifying this configuration will not take effect on old tags</td>
-</tr>
-<tr>
-<td>tag.auto-create.max-age-ms</td>
-<td>-1</td>
-<td>Time of automatically created Tag to retain, -1 means keep it forever. Modifying this configuration will not take effect on old tags</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>tag.auto-create.enabled</td>
+ <td>false</td>
+ <td>Enables automatically creating tags</td>
+ </tr>
+ <tr>
+ <td>tag.auto-create.trigger.period</td>
+ <td>daily</td>
+ <td>Period of creating tags, support <code>daily</code>,<code>hourly</code> now</td>
+ </tr>
+ <tr>
+ <td>tag.auto-create.trigger.offset.minutes</td>
+ <td>0</td>
+ <td>The minutes by which the tag is created after midnight (00:00)</td>
+ </tr>
+ <tr>
+ <td>tag.auto-create.trigger.max-delay.minutes</td>
+ <td>60</td>
+ <td>The maximum delay time for creating a tag</td>
+ </tr>
+ <tr>
+ <td>tag.auto-create.tag-format</td>
+ <td>’tag-‘yyyyMMdd for daily and ’tag-‘yyyyMMddHH for hourly periods</td>
+ <td>The format of the name for tag. Modifying this configuration will not take effect on old tags</td>
+ </tr>
+ <tr>
+ <td>tag.auto-create.max-age-ms</td>
+ <td>-1</td>
+ <td>Time of automatically created Tag to retain, -1 means keep it forever. Modifying this configuration will not take effect on old tags</td>
+ </tr>
+ </tbody>
</table>
<h2 id="mixed-format-configurations">Mixed Format configurations</h2>
<p>If using Iceberg Format，please refer to <a href="https://iceberg.apache.org/docs/latest/configuration/">Iceberg configurations</a>，the following configurations are only applicable to Mixed Format.</p>
<h3 id="reading-configurations">Reading configurations</h3>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>read.split.open-file-cost</td>
-<td>4194304(4MB)</td>
-<td>The estimated cost to open a file</td>
-</tr>
-<tr>
-<td>read.split.planning-lookback</td>
-<td>10</td>
-<td>Number of bins to consider when combining input splits</td>
-</tr>
-<tr>
-<td>read.split.target-size</td>
-<td>134217728(128MB)</td>
-<td>Target size when combining data input splits</td>
-</tr>
-<tr>
-<td>read.split.delete-ratio</td>
-<td>0.05</td>
-<td>When the ratio of delete files is below this threshold, the read task will be split into more tasks to improve query speed</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>read.split.open-file-cost</td>
+ <td>4194304(4MB)</td>
+ <td>The estimated cost to open a file</td>
+ </tr>
+ <tr>
+ <td>read.split.planning-lookback</td>
+ <td>10</td>
+ <td>Number of bins to consider when combining input splits</td>
+ </tr>
+ <tr>
+ <td>read.split.target-size</td>
+ <td>134217728(128MB)</td>
+ <td>Target size when combining data input splits</td>
+ </tr>
+ <tr>
+ <td>read.split.delete-ratio</td>
+ <td>0.05</td>
+ <td>When the ratio of delete files is below this threshold, the read task will be split into more tasks to improve query speed</td>
+ </tr>
+ </tbody>
</table>
<h3 id="writing-configurations">Writing configurations</h3>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>base.write.format</td>
-<td>parquet</td>
-<td>File format for the table for BaseStore, applicable to KeyedTable</td>
-</tr>
-<tr>
-<td>change.write.format</td>
-<td>parquet</td>
-<td>File format for the table for ChangeStore, applicable to KeyedTable</td>
-</tr>
-<tr>
-<td>write.format.default</td>
-<td>parquet</td>
-<td>Default file format for the table, applicable to UnkeyedTable</td>
-</tr>
-<tr>
-<td>base.file-index.hash-bucket</td>
-<td>4</td>
-<td>Initial number of buckets for BaseStore auto-bucket</td>
-</tr>
-<tr>
-<td>change.file-index.hash-bucket</td>
-<td>4</td>
-<td>Initial number of buckets for ChangeStore auto-bucket</td>
-</tr>
-<tr>
-<td>write.target-file-size-bytes</td>
-<td>134217728(128MB)</td>
-<td>Target size when writing</td>
-</tr>
-<tr>
-<td>write.upsert.enabled</td>
-<td>false</td>
-<td>Enable upsert mode, multiple insert data with the same primary key will be merged if enabled</td>
-</tr>
-<tr>
-<td>write.distribution-mode</td>
-<td>hash</td>
-<td>Shuffle rules for writing. UnkeyedTable can choose between none and hash, while KeyedTable can only choose hash</td>
-</tr>
-<tr>
-<td>write.distribution.hash-mode</td>
-<td>auto</td>
-<td>Auto-bucket mode, which supports primary-key, partition-key, primary-partition-key, and auto</td>
-</tr>
-<tr>
-<td>base.refresh-interval</td>
-<td>-1 (Closed)</td>
-<td>The interval for refreshing the BaseStore</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>base.write.format</td>
+ <td>parquet</td>
+ <td>File format for the table for BaseStore, applicable to KeyedTable</td>
+ </tr>
+ <tr>
+ <td>change.write.format</td>
+ <td>parquet</td>
+ <td>File format for the table for ChangeStore, applicable to KeyedTable</td>
+ </tr>
+ <tr>
+ <td>write.format.default</td>
+ <td>parquet</td>
+ <td>Default file format for the table, applicable to UnkeyedTable</td>
+ </tr>
+ <tr>
+ <td>base.file-index.hash-bucket</td>
+ <td>4</td>
+ <td>Initial number of buckets for BaseStore auto-bucket</td>
+ </tr>
+ <tr>
+ <td>change.file-index.hash-bucket</td>
+ <td>4</td>
+ <td>Initial number of buckets for ChangeStore auto-bucket</td>
+ </tr>
+ <tr>
+ <td>write.target-file-size-bytes</td>
+ <td>134217728(128MB)</td>
+ <td>Target size when writing</td>
+ </tr>
+ <tr>
+ <td>write.upsert.enabled</td>
+ <td>false</td>
+ <td>Enable upsert mode, multiple insert data with the same primary key will be merged if enabled</td>
+ </tr>
+ <tr>
+ <td>write.distribution-mode</td>
+ <td>hash</td>
+ <td>Shuffle rules for writing. UnkeyedTable can choose between none and hash, while KeyedTable can only choose hash</td>
+ </tr>
+ <tr>
+ <td>write.distribution.hash-mode</td>
+ <td>auto</td>
+ <td>Auto-bucket mode, which supports primary-key, partition-key, primary-partition-key, and auto</td>
+ </tr>
+ <tr>
+ <td>base.refresh-interval</td>
+ <td>-1 (Closed)</td>
+ <td>The interval for refreshing the BaseStore</td>
+ </tr>
+ </tbody>
</table>
<h3 id="logstore-configurations">LogStore configurations</h3>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>log-store.enabled</td>
-<td>false</td>
-<td>Enables LogStore</td>
-</tr>
-<tr>
-<td>log-store.type</td>
-<td>kafka</td>
-<td>Type of LogStore, which supports ‘kafka’ and ‘pulsar’</td>
-</tr>
-<tr>
-<td>log-store.address</td>
-<td>NULL</td>
-<td>Address of LogStore, required when LogStore enabled. For Kafka, this is the Kafka bootstrap servers. For Pulsar, this is the Pulsar Service URL, such as ‘pulsar://localhost:6650’</td>
-</tr>
-<tr>
-<td>log-store.topic</td>
-<td>NULL</td>
-<td>Topic of LogStore, required when LogStore enabled</td>
-</tr>
-<tr>
-<td>properties.pulsar.admin.adminUrl</td>
-<td>NULL</td>
-<td>HTTP URL of Pulsar admin, such as ‘<a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a>’. Only required when log-store.type=pulsar</td>
-</tr>
-<tr>
-<td>properties.XXX</td>
-<td>NULL</td>
-<td>Other configurations of LogStore. <br><br>For Kafka, all the configurations supported by Kafka Consumer/Producer can be set by prefixing them with <code>properties.</code>，<br>such as <code>'properties.batch.size'='16384'</code>，<br>refer to <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka Consumer Configurations</a>, <a href="https://kafka.apache.org/documentation/#producerconfigs">Kafka Producer Configurations</a> for more details.<br><br> For Pulsar，all the [...]
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>log-store.enabled</td>
+ <td>false</td>
+ <td>Enables LogStore</td>
+ </tr>
+ <tr>
+ <td>log-store.type</td>
+ <td>kafka</td>
+ <td>Type of LogStore, which supports ‘kafka’ and ‘pulsar’</td>
+ </tr>
+ <tr>
+ <td>log-store.address</td>
+ <td>NULL</td>
+ <td>Address of LogStore, required when LogStore enabled. For Kafka, this is the Kafka bootstrap servers. For Pulsar, this is the Pulsar Service URL, such as ‘pulsar://localhost:6650’</td>
+ </tr>
+ <tr>
+ <td>log-store.topic</td>
+ <td>NULL</td>
+ <td>Topic of LogStore, required when LogStore enabled</td>
+ </tr>
+ <tr>
+ <td>properties.pulsar.admin.adminUrl</td>
+ <td>NULL</td>
+ <td>HTTP URL of Pulsar admin, such as ‘<a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a>’. Only required when log-store.type=pulsar</td>
+ </tr>
+ <tr>
+ <td>properties.XXX</td>
+ <td>NULL</td>
+ <td>Other configurations of LogStore. <br><br>For Kafka, all the configurations supported by Kafka Consumer/Producer can be set by prefixing them with <code>properties.</code>，<br>such as <code>'properties.batch.size'='16384'</code>，<br>refer to <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka Consumer Configurations</a>, <a href="https://kafka.apache.org/documentation/#producerconfigs">Kafka Producer Configurations</a> for more details.<br><br> For Pulsa [...]
+ </tr>
+ </tbody>
</table>
<h3 id="watermark-configurations">Watermark configurations</h3>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>table.event-time-field</td>
-<td>_ingest_time</td>
-<td>The event time field for calculating the watermark. The default <code>_ingest_time</code> indicates calculating with the time when the data was written</td>
-</tr>
-<tr>
-<td>table.watermark-allowed-lateness-second</td>
-<td>0</td>
-<td>The allowed lateness time in seconds when calculating watermark</td>
-</tr>
-<tr>
-<td>table.event-time-field.datetime-string-format</td>
-<td><code>yyyy-MM-dd HH:mm:ss</code></td>
-<td>The format of event time when it is in string format</td>
-</tr>
-<tr>
-<td>table.event-time-field.datetime-number-format</td>
-<td>TIMESTAMP_MS</td>
-<td>The format of event time when it is in numeric format, which supports TIMESTAMP_MS (timestamp in milliseconds) and TIMESTAMP_S (timestamp in seconds)</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>table.event-time-field</td>
+ <td>_ingest_time</td>
+ <td>The event time field for calculating the watermark. The default <code>_ingest_time</code> indicates calculating with the time when the data was written</td>
+ </tr>
+ <tr>
+ <td>table.watermark-allowed-lateness-second</td>
+ <td>0</td>
+ <td>The allowed lateness time in seconds when calculating watermark</td>
+ </tr>
+ <tr>
+ <td>table.event-time-field.datetime-string-format</td>
+ <td><code>yyyy-MM-dd HH:mm:ss</code></td>
+ <td>The format of event time when it is in string format</td>
+ </tr>
+ <tr>
+ <td>table.event-time-field.datetime-number-format</td>
+ <td>TIMESTAMP_MS</td>
+ <td>The format of event time when it is in numeric format, which supports TIMESTAMP_MS (timestamp in milliseconds) and TIMESTAMP_S (timestamp in seconds)</td>
+ </tr>
+ </tbody>
</table>
<h3 id="mixed-hive-format-configurations">Mixed-Hive format configurations</h3>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>base.hive.auto-sync-schema-change</td>
-<td>true</td>
-<td>Whether synchronize schema changes of Hive Table from HMS</td>
-</tr>
-<tr>
-<td>base.hive.auto-sync-data-write</td>
-<td>false</td>
-<td>Whether synchronize data changes of Hive Table from HMS, this should be true when writing to Hive</td>
-</tr>
-<tr>
-<td>base.hive.consistent-write.enabled</td>
-<td>true</td>
-<td>To avoid writing dirty data, the files written to the Hive directory will be hidden files and renamed to visible files upon commit.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>base.hive.auto-sync-schema-change</td>
+ <td>true</td>
+ <td>Whether synchronize schema changes of Hive Table from HMS</td>
+ </tr>
+ <tr>
+ <td>base.hive.auto-sync-data-write</td>
+ <td>false</td>
+ <td>Whether synchronize data changes of Hive Table from HMS, this should be true when writing to Hive</td>
+ </tr>
+ <tr>
+ <td>base.hive.consistent-write.enabled</td>
+ <td>true</td>
+ <td>To avoid writing dirty data, the files written to the Hive directory will be hidden files and renamed to visible files upon commit.</td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/deployment-on-kubernetes/index.html b/output/docs/latest/deployment-on-kubernetes/index.html
index 5603fe2c3..b2c152dd9 100644
--- a/output/docs/latest/deployment-on-kubernetes/index.html
+++ b/output/docs/latest/deployment-on-kubernetes/index.html
@@ -763,7 +763,7 @@ You can modify the container configuration by changing the <code>optimizer.Kuber
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> cpu: 2
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> requests:
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> memory: 2048Mi
-</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> cpu: 2</span>
+</span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> cpu: 2</span>
</span></span></code></pre></div><h3 id="configure-the-rbac">Configure the RBAC</h3>
<p>By default, Helm Chart creates a service account, role, and role bind for Amoro deploy.
You can also modify this configuration to use an existing account.</p>
diff --git a/output/docs/latest/deployment/index.html b/output/docs/latest/deployment/index.html
index 9d2a426d8..9e6d5aed5 100644
--- a/output/docs/latest/deployment/index.html
+++ b/output/docs/latest/deployment/index.html
@@ -634,7 +634,7 @@
<p>You can choose to download the stable release package from <a href="../../../download/">download page</a>, or the source code form <a href="https://github.com/apache/amoro">Github</a> and compile it according to the README.</p>
<h2 id="system-requirements">System requirements</h2>
<ul>
-<li>Java 8 is required.</li>
+<li>Java 11 is required.</li>
<li>Optional: A RDBMS (PostgreSQL 14.x or higher, MySQL 5.5 or higher)</li>
<li>Optional: ZooKeeper 3.4.x or higher</li>
</ul>
@@ -758,40 +758,40 @@ The configuration for kyuubi mode can refer to: <a href="../using-kyuubi/">Using
</span></span><span style="display:flex;"><span> <span style="color:#f92672">local.using-session-catalog-for-hive</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>More properties the terminal supports including:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>terminal.backend</td>
-<td>local</td>
-<td>Terminal backend implementation. local, kyuubi and custom are valid values.</td>
-</tr>
-<tr>
-<td>terminal.factory</td>
-<td>-</td>
-<td>Session factory implement of terminal, <code>terminal.backend</code> must be <code>custom</code> if this is set.</td>
-</tr>
-<tr>
-<td>terminal.result.limit</td>
-<td>1000</td>
-<td>Row limit of result-set</td>
-</tr>
-<tr>
-<td>terminal.stop-on-error</td>
-<td>false</td>
-<td>When a statement fails to execute, stop execution or continue executing the remaining statements.</td>
-</tr>
-<tr>
-<td>terminal.session.timeout</td>
-<td>30</td>
-<td>Session timeout in minutes.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>terminal.backend</td>
+ <td>local</td>
+ <td>Terminal backend implementation. local, kyuubi and custom are valid values.</td>
+ </tr>
+ <tr>
+ <td>terminal.factory</td>
+ <td>-</td>
+ <td>Session factory implement of terminal, <code>terminal.backend</code> must be <code>custom</code> if this is set.</td>
+ </tr>
+ <tr>
+ <td>terminal.result.limit</td>
+ <td>1000</td>
+ <td>Row limit of result-set</td>
+ </tr>
+ <tr>
+ <td>terminal.stop-on-error</td>
+ <td>false</td>
+ <td>When a statement fails to execute, stop execution or continue executing the remaining statements.</td>
+ </tr>
+ <tr>
+ <td>terminal.session.timeout</td>
+ <td>30</td>
+ <td>Session timeout in minutes.</td>
+ </tr>
+ </tbody>
</table>
<h3 id="configure-metric-reporter">Configure metric reporter</h3>
<p>Amoro provides metric reporters by plugin mechanism to connect to external metric systems.</p>
@@ -846,59 +846,59 @@ Currently, AMS provides built-in support for base64 decryption, and users can al
<p>The following environment variables take effect during the startup process of AMS,
you can set up those environments to overwrite the default value.</p>
<table>
-<thead>
-<tr>
-<th>Environments variable name</th>
-<th>Default value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>AMORO_CONF_DIR</td>
-<td>${AMORO_HOME}/conf</td>
-<td>location where Amoro loading config files.</td>
-</tr>
-<tr>
-<td>AMORO_LOG_DIR</td>
-<td>${AMORO_HOME}/logs</td>
-<td>location where the logs files output</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Environments variable name</th>
+ <th>Default value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>AMORO_CONF_DIR</td>
+ <td>${AMORO_HOME}/conf</td>
+ <td>location where Amoro loading config files.</td>
+ </tr>
+ <tr>
+ <td>AMORO_LOG_DIR</td>
+ <td>${AMORO_HOME}/logs</td>
+ <td>location where the logs files output</td>
+ </tr>
+ </tbody>
</table>
<p>Note: <code>$AMORO_HOME</code> can’t be overwritten from environment variable. It always points to the parent dir of <code>./bin</code>.</p>
<h3 id="configure-ams-jvm">Configure AMS JVM</h3>
<p>The following JVM options could be set in <code>${AMORO_CONF_DIR}/jvm.properties</code>.</p>
<table>
-<thead>
-<tr>
-<th>Property Name</th>
-<th>Related Jvm option</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>xms</td>
-<td>“-Xms${value}m</td>
-<td>Xms config for jvm</td>
-</tr>
-<tr>
-<td>xmx</td>
-<td>“-Xmx${value}m</td>
-<td>Xmx config for jvm</td>
-</tr>
-<tr>
-<td>jmx.remote.port</td>
-<td>“-Dcom.sun.management.jmxremote.port=${value}</td>
-<td>Enable remote debug</td>
-</tr>
-<tr>
-<td>extra.options</td>
-<td>“JAVA_OPTS="${JAVA_OPTS} ${JVM_EXTRA_CONFIG}”</td>
-<td>The addition jvm options</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property Name</th>
+ <th>Related Jvm option</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>xms</td>
+ <td>“-Xms${value}m</td>
+ <td>Xms config for jvm</td>
+ </tr>
+ <tr>
+ <td>xmx</td>
+ <td>“-Xmx${value}m</td>
+ <td>Xmx config for jvm</td>
+ </tr>
+ <tr>
+ <td>jmx.remote.port</td>
+ <td>“-Dcom.sun.management.jmxremote.port=${value}</td>
+ <td>Enable remote debug</td>
+ </tr>
+ <tr>
+ <td>extra.options</td>
+ <td>“JAVA_OPTS="${JAVA_OPTS} ${JVM_EXTRA_CONFIG}”</td>
+ <td>The addition jvm options</td>
+ </tr>
+ </tbody>
</table>
<h2 id="start-ams">Start AMS</h2>
<p>Enter the directory amoro-x.y.z and execute bin/ams.sh start to start AMS.</p>
diff --git a/output/docs/latest/engines/index.xml b/output/docs/latest/engines/index.xml
index 7a8497580..0d66333d2 100644
--- a/output/docs/latest/engines/index.xml
+++ b/output/docs/latest/engines/index.xml
@@ -12,84 +12,84 @@
<link>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</guid>
- <description>Apache CDC Ingestion CDC stands for Change Data Capture, which is a broad concept, as long as it can capture the change data, it can be called CDC. Flink CDC is a Log message-based data capture tool, all the inventory and incremental data can be captured. Taking MySQL as an example, it can easily capture Binlog data through Debezium and process the calculations in real time to send them to the data lake.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DataStream</title>
<link>https://amoro.apache.org/docs/latest/flink-datastream/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-datastream/</guid>
- <description>Flink DataStream Add maven dependency To add a dependency on Mixed-format flink connector in Maven, add the following to your pom.xml:
&lt;dependencies&gt; ... &lt;dependency&gt; &lt;groupId&gt;org.apache.amoro&lt;/groupId&gt; &lt;!-- For example: amoro-format-mixed-flink-runtime-1.15 --&gt; &lt;artifactId&gt;amoro-format-mixed-flink-runtime-${flink.minor-version}&lt;/artifactId&gt; &lt;!-- For example: 0. [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DDL</title>
<link>https://amoro.apache.org/docs/latest/flink-ddl/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-ddl/</guid>
- <description>Flink DDL Create catalogs Flink SQL The following statement can be executed to create a Flink catalog:
CREATE CATALOG &lt;catalog_name&gt; WITH ( &#39;type&#39;=&#39;mixed_iceberg&#39;, `&lt;config_key&gt;`=`&lt;config_value&gt;` ); Where &lt;catalog_name&gt; is the user-defined name of the Flink catalog, and &lt;config_key&gt;=&lt;config_value&gt; has the following configurations:
Key Default Valu [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DML</title>
<link>https://amoro.apache.org/docs/latest/flink-dml/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-dml/</guid>
- <description>Flink DML Querying with SQL Amoro tables support reading data in stream or batch mode through Flink SQL. You can switch modes using the following methods:
-- Run Flink tasks in streaming mode in the current session SET execution.runtime-mode = streaming; -- Run Flink tasks in batch mode in the current session SET execution.runtime-mode = batch; The following Hint Options are supported:
Key Default Value Type Required Description source.parallelism (none) Intege [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink Getting Started</title>
<link>https://amoro.apache.org/docs/latest/flink-getting-started/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-getting-started/</guid>
- <description>Flink Getting Started Iceberg format The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at Iceberg Flink user manual for more information.
Paimon format The Paimon Format can be accessed using the Connector provided by Paimon. Refer to the documentation at Paimon Flink user manual for more information.
Mixed format The Apache Flink engine can process Amoro table data in batch and streaming mode.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Configuration</title>
<link>https://amoro.apache.org/docs/latest/spark-configuration/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-configuration/</guid>
- <description>Spark Configuration Catalogs configuration Using Mixed-Format in a standalone catalog Starting from version 3.x, Spark supports configuring an independent Catalog. If you want to use a Mixed-Format table in a standalone Catalog, you can create a mixed_catalog and load catalog metadata from AMS with following properties:
spark.sql.catalog.mixed_catalog=org.apache.amoro.spark.MixedFormatSparkCatalog spark.sql.catalog.mixed_catalog.ams.uri=thrift://${AMS_HOST}:${AMS_P [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark DDL</title>
<link>https://amoro.apache.org/docs/latest/spark-ddl/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-ddl/</guid>
- <description>Spark DDL CREATE TABLE To create an MixedFormat table under an Amoro Catalog, you can use using mixed_iceberg or using mixed_hive to specify the provider in the CREATE TABLE statement. If the Catalog type is Hive, the created table will be a Hive-compatible table.
CREATE TABLE mixed_catalog.db.sample ( id bigint COMMENT &#34;unique id&#34;, data string ) USING mixed_iceberg PRIMARY KEY You can use PRIMARY KEY in the CREATE TABLE statement to specify the pri [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Getting Started</title>
<link>https://amoro.apache.org/docs/latest/spark-getting-started/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-getting-started/</guid>
- <description>Spark Getting Started Iceberg Format The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at Iceberg Spark Connector for more information.
Paimon Format The Paimon Format can be accessed using the Connector provided by Paimon. Refer to the documentation at Paimon Spark Connector for more information.
Mixed Format To use Amoro in a Spark shell, use the &ndash;packages option:
spark-shell --packages org.ap [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Queries</title>
<link>https://amoro.apache.org/docs/latest/spark-queries/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-queries/</guid>
- <description>Spark Queries Querying with SQL Querying Mixed-Format table by merge on read Using Select statement to query on Mixed-Format tables.
SELECT * FROM mixed_catalog.db.sample The Mixed-Format connector will merge the data from BaseStore and ChangeStore.
Query on change store For a Mixed-Format table with primary keys. you can query on ChangeStore by .change.
SELECT * FROM mixed_catalog.db.sample.change +---+----+----+---------------+------------+--------------+ [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Writes</title>
<link>https://amoro.apache.org/docs/latest/spark-writes/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-writes/</guid>
- <description>Spark Writes Writing with SQL INSERT OVERWRITE INSERT OVERWRITE can replace the partition in a table with the results of a query.
The default overwrite mode of Spark is Static, you can change the overwrite mode by
SET spark.sql.sources.partitionOverwriteMode=dynamic To demonstrate the behavior of dynamic and static overwrites, a test table is defined using the following DDL:
CREATE TABLE mixed_catalog.db.sample ( id int, data string, ts timestamp, primary k [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Trino</title>
<link>https://amoro.apache.org/docs/latest/trino/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/trino/</guid>
- <description>Trino Iceberg format Iceberg format can be accessed using the Iceberg Connector provided by Trino. please refer to the documentation at Iceberg Trino user manual for more information.
Paimon format Paimon format can be accessed using the Paimon Connector provided by Trino. please refer to the documentation at Paimon Trino user manual for more information.
Mixed format Install Create the {trino_home}/plugin/amoro directory in the Trino installation package, and [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Logstore</title>
<link>https://amoro.apache.org/docs/latest/flink-using-logstore/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-using-logstore/</guid>
- <description>Using Logstore Due to the limitations of traditional offline data warehouse architectures in supporting real-time business needs, real-time data warehousing has experienced rapid evolution in recent years. In the architecture of real-time data warehousing, Apache Kafka is often used as the storage system for real-time data. However, this also brings about the issue of data disconnection between offline data warehouses.
Developers often need to pay attention to data [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/flink-datastream/index.html b/output/docs/latest/flink-datastream/index.html
index ae3d1b668..20368f41d 100644
--- a/output/docs/latest/flink-datastream/index.html
+++ b/output/docs/latest/flink-datastream/index.html
@@ -694,52 +694,52 @@
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The map properties contain below keys, <strong>currently only valid for non-primary key tables</strong>:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>case-sensitive</td>
-<td>false</td>
-<td>Boolean</td>
-<td>No</td>
-<td>Case-sensitive</td>
-</tr>
-<tr>
-<td>snapshot-id</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>Read the full amount of data of the specified snapshot, only effective when streaming is false or not configured</td>
-</tr>
-<tr>
-<td>as-of-timestamp</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Read the last time less than the timestamp The full amount of snapshot data is valid only when streaming is false or not configured</td>
-</tr>
-<tr>
-<td>start-snapshot-id</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>When streaming is false, end-snapshot-id needs to be used to read the two intervals Incremental data (snapshot1, snapshot2]. When streaming is true, read the incremental data after the snapshot, if not specified, read the incremental data after the current snapshot (not including the current one)</td>
-</tr>
-<tr>
-<td>end-snapshot-id</td>
-<td>(none )</td>
-<td>String</td>
-<td>No</td>
-<td>Need to cooperate with start-snapshot-id to read incremental data in two intervals (snapshot1, snapshot2]</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>case-sensitive</td>
+ <td>false</td>
+ <td>Boolean</td>
+ <td>No</td>
+ <td>Case-sensitive</td>
+ </tr>
+ <tr>
+ <td>snapshot-id</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>Read the full amount of data of the specified snapshot, only effective when streaming is false or not configured</td>
+ </tr>
+ <tr>
+ <td>as-of-timestamp</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Read the last time less than the timestamp The full amount of snapshot data is valid only when streaming is false or not configured</td>
+ </tr>
+ <tr>
+ <td>start-snapshot-id</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>When streaming is false, end-snapshot-id needs to be used to read the two intervals Incremental data (snapshot1, snapshot2]. When streaming is true, read the incremental data after the snapshot, if not specified, read the incremental data after the current snapshot (not including the current one)</td>
+ </tr>
+ <tr>
+ <td>end-snapshot-id</td>
+ <td>(none )</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Need to cooperate with start-snapshot-id to read incremental data in two intervals (snapshot1, snapshot2]</td>
+ </tr>
+ </tbody>
</table>
<h3 id="streaming-mode">Streaming mode</h3>
<p>Amoro supports reading incremental data in FileStore or LogStore through Java API in Streaming mode</p>
diff --git a/output/docs/latest/flink-ddl/index.html b/output/docs/latest/flink-ddl/index.html
index a8c53040b..5b653bc35 100644
--- a/output/docs/latest/flink-ddl/index.html
+++ b/output/docs/latest/flink-ddl/index.html
@@ -640,109 +640,109 @@
</span></span><span style="display:flex;"><span>);
</span></span></code></pre></div><p>Where <code><catalog_name></code> is the user-defined name of the Flink catalog, and <code><config_key></code>=<code><config_value></code> has the following configurations:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>type</td>
-<td>N/A</td>
-<td>String</td>
-<td>Yes</td>
-<td>Catalog type, validate values are mixed_iceberg and mixed_hive</td>
-</tr>
-<tr>
-<td>ams.uri</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>The URI for Amoro Metastore is thrift://<code><ip></code>:<code><port></code>/<code><catalog_name_in_metastore></code>.<br>If high availability is enabled for AMS, it can also be specified in the form of zookeeper://{zookeeper-server}/{cluster-name}/{catalog-name}.</td>
-</tr>
-<tr>
-<td>default-database</td>
-<td>default</td>
-<td>String</td>
-<td>No</td>
-<td>The default database to use</td>
-</tr>
-<tr>
-<td>property-version</td>
-<td>1</td>
-<td>Integer</td>
-<td>No</td>
-<td>Catalog properties version, this option is for future backward compatibility</td>
-</tr>
-<tr>
-<td>catalog-type</td>
-<td>N/A</td>
-<td>String</td>
-<td>No</td>
-<td>Metastore type of the catalog, validate values are hadoop, hive, rest, custom</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>type</td>
+ <td>N/A</td>
+ <td>String</td>
+ <td>Yes</td>
+ <td>Catalog type, validate values are mixed_iceberg and mixed_hive</td>
+ </tr>
+ <tr>
+ <td>ams.uri</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The URI for Amoro Metastore is thrift://<code><ip></code>:<code><port></code>/<code><catalog_name_in_metastore></code>.<br>If high availability is enabled for AMS, it can also be specified in the form of zookeeper://{zookeeper-server}/{cluster-name}/{catalog-name}.</td>
+ </tr>
+ <tr>
+ <td>default-database</td>
+ <td>default</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The default database to use</td>
+ </tr>
+ <tr>
+ <td>property-version</td>
+ <td>1</td>
+ <td>Integer</td>
+ <td>No</td>
+ <td>Catalog properties version, this option is for future backward compatibility</td>
+ </tr>
+ <tr>
+ <td>catalog-type</td>
+ <td>N/A</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Metastore type of the catalog, validate values are hadoop, hive, rest, custom</td>
+ </tr>
+ </tbody>
</table>
<p>The authentication information of AMS catalog can upload configuration files on AMS website,
or specify the authentication information and configuration file paths when creating catalogs with Flink DDL</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>properties.auth.load-from-ams</td>
-<td>True</td>
-<td>BOOLEAN</td>
-<td>No</td>
-<td>Whether to load security verification configuration from AMS. <br>True: load from AMS; <br>false: do not use AMS configuration. <br>Note: regardless of whether this parameter is configured, as long as the user has configured the auth.*** related parameters below, this configuration will be used for access.</td>
-</tr>
-<tr>
-<td>properties.auth.type</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Table security verification type, valid values: simple, kerberos, or not configured. Default not configured, no permission check is required. simple: use the hadoop username, used in conjunction with the parameter ‘properties.auth.simple.hadoop_username’; kerberos: configure kerberos permission verification, used in conjunction with the parameters ‘properties.auth.kerberos.principal’, ‘properties.auth.kerberos.keytab’, ‘properties.auth.kerber [...]
-</tr>
-<tr>
-<td>properties.auth.simple.hadoop_username</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Access using this hadoop username, required when ‘properties.auth.type’=‘simple’.</td>
-</tr>
-<tr>
-<td>properties.auth.kerberos.principal</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Configuration of kerberos principal, required when ‘properties.auth.type’=‘kerberos’.</td>
-</tr>
-<tr>
-<td>properties.auth.kerberos.krb.path</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>The absolute path to the krb5.conf configuration file for kerberos (the local file path of the Flink SQL submission machine, if the SQL task is submitted with the Flink SQL Client, the path is the local path of the same node, e.g. /XXX/XXX/krb5.conf).’ required if ‘properties.auth.type’ = ‘kerberos’.</td>
-</tr>
-<tr>
-<td>properties.auth.kerberos.keytab.path</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>The absolute path to the XXX.keytab configuration file for kerberos (the local file path of the Flink SQL submission machine, if the SQL task is submitted with the Flink SQL Client, the path is the local path of the same node, e.g. /XXX/XXX/XXX.keytab).’ required if ‘properties.auth.type’ = ‘kerberos’.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>properties.auth.load-from-ams</td>
+ <td>True</td>
+ <td>BOOLEAN</td>
+ <td>No</td>
+ <td>Whether to load security verification configuration from AMS. <br>True: load from AMS; <br>false: do not use AMS configuration. <br>Note: regardless of whether this parameter is configured, as long as the user has configured the auth.*** related parameters below, this configuration will be used for access.</td>
+ </tr>
+ <tr>
+ <td>properties.auth.type</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Table security verification type, valid values: simple, kerberos, or not configured. Default not configured, no permission check is required. simple: use the hadoop username, used in conjunction with the parameter ‘properties.auth.simple.hadoop_username’; kerberos: configure kerberos permission verification, used in conjunction with the parameters ‘properties.auth.kerberos.principal’, ‘properties.auth.kerberos.keytab’, ‘properties.a [...]
+ </tr>
+ <tr>
+ <td>properties.auth.simple.hadoop_username</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Access using this hadoop username, required when ‘properties.auth.type’=‘simple’.</td>
+ </tr>
+ <tr>
+ <td>properties.auth.kerberos.principal</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Configuration of kerberos principal, required when ‘properties.auth.type’=‘kerberos’.</td>
+ </tr>
+ <tr>
+ <td>properties.auth.kerberos.krb.path</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The absolute path to the krb5.conf configuration file for kerberos (the local file path of the Flink SQL submission machine, if the SQL task is submitted with the Flink SQL Client, the path is the local path of the same node, e.g. /XXX/XXX/krb5.conf).’ required if ‘properties.auth.type’ = ‘kerberos’.</td>
+ </tr>
+ <tr>
+ <td>properties.auth.kerberos.keytab.path</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The absolute path to the XXX.keytab configuration file for kerberos (the local file path of the Flink SQL submission machine, if the SQL task is submitted with the Flink SQL Client, the path is the local path of the same node, e.g. /XXX/XXX/XXX.keytab).’ required if ‘properties.auth.type’ = ‘kerberos’.</td>
+ </tr>
+ </tbody>
</table>
<h3 id="yaml-configuration">YAML configuration</h3>
<p>Refer to the Flink SQL Client <a href="https://nightlies.apache.org/flink/flink-docs-release-1.12/dev/table/sqlClient.html#environment-files">official configuration</a>.
@@ -837,177 +837,177 @@ Modify the <code>conf/sql-client-defaults.yaml</code> file in the Flink director
<h2 id="supported-types">Supported types</h2>
<h3 id="mixed-hive-data-types">Mixed-Hive data types</h3>
<table>
-<thead>
-<tr>
-<th>Flink Data Type</th>
-<th>Hive Data Type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>STRING</td>
-<td>CHAR(p)</td>
-</tr>
-<tr>
-<td>STRING</td>
-<td>VARCHAR(p)</td>
-</tr>
-<tr>
-<td>STRING</td>
-<td>STRING</td>
-</tr>
-<tr>
-<td>BOOLEAN</td>
-<td>BOOLEAN</td>
-</tr>
-<tr>
-<td>INT</td>
-<td>TINYINT</td>
-</tr>
-<tr>
-<td>INT</td>
-<td>SMALLINT</td>
-</tr>
-<tr>
-<td>INT</td>
-<td>INT</td>
-</tr>
-<tr>
-<td>BIGINT</td>
-<td>BIGINT</td>
-</tr>
-<tr>
-<td>FLOAT</td>
-<td>FLOAT</td>
-</tr>
-<tr>
-<td>DOUBLE</td>
-<td>DOUBLE</td>
-</tr>
-<tr>
-<td>DECIMAL(p, s)</td>
-<td>DECIMAL(p, s)</td>
-</tr>
-<tr>
-<td>DATE</td>
-<td>DATE</td>
-</tr>
-<tr>
-<td>TIMESTAMP(6)</td>
-<td>TIMESTAMP</td>
-</tr>
-<tr>
-<td>VARBINARY</td>
-<td>BINARY</td>
-</tr>
-<tr>
-<td>ARRAY<T></td>
-<td>ARRAY<T></td>
-</tr>
-<tr>
-<td>MAP<K, V></td>
-<td>MAP<K, V></td>
-</tr>
-<tr>
-<td>ROW</td>
-<td>STRUCT</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Flink Data Type</th>
+ <th>Hive Data Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>STRING</td>
+ <td>CHAR(p)</td>
+ </tr>
+ <tr>
+ <td>STRING</td>
+ <td>VARCHAR(p)</td>
+ </tr>
+ <tr>
+ <td>STRING</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>BOOLEAN</td>
+ <td>BOOLEAN</td>
+ </tr>
+ <tr>
+ <td>INT</td>
+ <td>TINYINT</td>
+ </tr>
+ <tr>
+ <td>INT</td>
+ <td>SMALLINT</td>
+ </tr>
+ <tr>
+ <td>INT</td>
+ <td>INT</td>
+ </tr>
+ <tr>
+ <td>BIGINT</td>
+ <td>BIGINT</td>
+ </tr>
+ <tr>
+ <td>FLOAT</td>
+ <td>FLOAT</td>
+ </tr>
+ <tr>
+ <td>DOUBLE</td>
+ <td>DOUBLE</td>
+ </tr>
+ <tr>
+ <td>DECIMAL(p, s)</td>
+ <td>DECIMAL(p, s)</td>
+ </tr>
+ <tr>
+ <td>DATE</td>
+ <td>DATE</td>
+ </tr>
+ <tr>
+ <td>TIMESTAMP(6)</td>
+ <td>TIMESTAMP</td>
+ </tr>
+ <tr>
+ <td>VARBINARY</td>
+ <td>BINARY</td>
+ </tr>
+ <tr>
+ <td>ARRAY<T></td>
+ <td>ARRAY<T></td>
+ </tr>
+ <tr>
+ <td>MAP<K, V></td>
+ <td>MAP<K, V></td>
+ </tr>
+ <tr>
+ <td>ROW</td>
+ <td>STRUCT</td>
+ </tr>
+ </tbody>
</table>
<h3 id="mixed_iceberg-data-types">mixed_iceberg data types</h3>
<table>
-<thead>
-<tr>
-<th>Flink Data Type</th>
-<th>Mixed-Iceberg Data Type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>CHAR(p)</td>
-<td>STRING</td>
-</tr>
-<tr>
-<td>VARCHAR(p)</td>
-<td>STRING</td>
-</tr>
-<tr>
-<td>STRING</td>
-<td>STRING</td>
-</tr>
-<tr>
-<td>BOOLEAN</td>
-<td>BOOLEAN</td>
-</tr>
-<tr>
-<td>TINYINT</td>
-<td>INT</td>
-</tr>
-<tr>
-<td>SMALLINT</td>
-<td>INT</td>
-</tr>
-<tr>
-<td>INT</td>
-<td>INT</td>
-</tr>
-<tr>
-<td>BIGINT</td>
-<td>LONG</td>
-</tr>
-<tr>
-<td>FLOAT</td>
-<td>FLOAT</td>
-</tr>
-<tr>
-<td>DOUBLE</td>
-<td>DOUBLE</td>
-</tr>
-<tr>
-<td>DECIMAL(p, s)</td>
-<td>DECIMAL(p, s)</td>
-</tr>
-<tr>
-<td>DATE</td>
-<td>DATE</td>
-</tr>
-<tr>
-<td>TIMESTAMP(6)</td>
-<td>TIMESTAMP</td>
-</tr>
-<tr>
-<td>TIMESTAMP(6) WITH LOCAL TIME ZONE</td>
-<td>TIMESTAMPTZ</td>
-</tr>
-<tr>
-<td>BINARY(p)</td>
-<td>FIXED(p)</td>
-</tr>
-<tr>
-<td>BINARY(16)</td>
-<td>UUID</td>
-</tr>
-<tr>
-<td>VARBINARY</td>
-<td>BINARY</td>
-</tr>
-<tr>
-<td>ARRAY<T></td>
-<td>ARRAY<T></td>
-</tr>
-<tr>
-<td>MAP<K, V></td>
-<td>MAP<K, V></td>
-</tr>
-<tr>
-<td>ROW</td>
-<td>STRUCT</td>
-</tr>
-<tr>
-<td>MULTISET<T></td>
-<td>MAP<T, INT></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Flink Data Type</th>
+ <th>Mixed-Iceberg Data Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>CHAR(p)</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>VARCHAR(p)</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>STRING</td>
+ <td>STRING</td>
+ </tr>
+ <tr>
+ <td>BOOLEAN</td>
+ <td>BOOLEAN</td>
+ </tr>
+ <tr>
+ <td>TINYINT</td>
+ <td>INT</td>
+ </tr>
+ <tr>
+ <td>SMALLINT</td>
+ <td>INT</td>
+ </tr>
+ <tr>
+ <td>INT</td>
+ <td>INT</td>
+ </tr>
+ <tr>
+ <td>BIGINT</td>
+ <td>LONG</td>
+ </tr>
+ <tr>
+ <td>FLOAT</td>
+ <td>FLOAT</td>
+ </tr>
+ <tr>
+ <td>DOUBLE</td>
+ <td>DOUBLE</td>
+ </tr>
+ <tr>
+ <td>DECIMAL(p, s)</td>
+ <td>DECIMAL(p, s)</td>
+ </tr>
+ <tr>
+ <td>DATE</td>
+ <td>DATE</td>
+ </tr>
+ <tr>
+ <td>TIMESTAMP(6)</td>
+ <td>TIMESTAMP</td>
+ </tr>
+ <tr>
+ <td>TIMESTAMP(6) WITH LOCAL TIME ZONE</td>
+ <td>TIMESTAMPTZ</td>
+ </tr>
+ <tr>
+ <td>BINARY(p)</td>
+ <td>FIXED(p)</td>
+ </tr>
+ <tr>
+ <td>BINARY(16)</td>
+ <td>UUID</td>
+ </tr>
+ <tr>
+ <td>VARBINARY</td>
+ <td>BINARY</td>
+ </tr>
+ <tr>
+ <td>ARRAY<T></td>
+ <td>ARRAY<T></td>
+ </tr>
+ <tr>
+ <td>MAP<K, V></td>
+ <td>MAP<K, V></td>
+ </tr>
+ <tr>
+ <td>ROW</td>
+ <td>STRUCT</td>
+ </tr>
+ <tr>
+ <td>MULTISET<T></td>
+ <td>MAP<T, INT></td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/flink-dml/index.html b/output/docs/latest/flink-dml/index.html
index 171d97bfc..12b8d688d 100644
--- a/output/docs/latest/flink-dml/index.html
+++ b/output/docs/latest/flink-dml/index.html
@@ -634,30 +634,30 @@
<h2 id="querying-with-sql">Querying with SQL</h2>
<p>Amoro tables support reading data in stream or batch mode through Flink SQL. You can switch modes using the following methods:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in streaming mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in batch mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> batch;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> batch;
</span></span></code></pre></div><p>The following Hint Options are supported:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>source.parallelism</td>
-<td>(none)</td>
-<td>Integer</td>
-<td>No</td>
-<td>Defines a custom parallelism for the source. By default, if this option is not defined, the planner will derive the parallelism for each statement individually by also considering the global configuration.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>source.parallelism</td>
+ <td>(none)</td>
+ <td>Integer</td>
+ <td>No</td>
+ <td>Defines a custom parallelism for the source. By default, if this option is not defined, the planner will derive the parallelism for each statement individually by also considering the global configuration.</td>
+ </tr>
+ </tbody>
</table>
<h3 id="batch-mode">Batch mode</h3>
<p>Use batch mode to read full and incremental data from FileStore.</p>
@@ -666,148 +666,148 @@
<p>LogStore does not support bounded reading.</p>
</blockquote>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in batch mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> batch;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> batch;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Enable dynamic table parameter configuration to make hint options configured in Flink SQL effective
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span>;
</span></span></code></pre></div><h3 id="batch-mode-non-primary-key-table">Batch mode (non-primary key table)</h3>
<p>Non-primary key tables support reading full data in batch mode, specifying snapshot data with snapshot-id or timestamp, and specifying the incremental data of the snapshot interval.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Read full data
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('streaming'='false')*/</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('streaming'='false')*/</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Read specified snapshot data
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('snapshot-id'='4411985347497777546')*/</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('snapshot-id'='4411985347497777546')*/</span>;
</span></span></code></pre></div><p>The supported parameters for bounded reads of non-primary-key tables in BaseStore include:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>snapshot-id<img width=100/></td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>Reading the full data of a specific snapshot only works when streaming is set to false.</td>
-</tr>
-<tr>
-<td>as-of-timestamp</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>Reading the full data of the latest snapshot taken before the specified timestamp only works when streaming is set to false.</td>
-</tr>
-<tr>
-<td>start-snapshot-id</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>When streaming is set to false, you need to specify the end-snapshot-id to read the incremental data within two intervals (snapshot1, snapshot2]. When streaming is set to true, you can read the incremental data after the specified snapshot. If not specified, it will read the incremental data after the current snapshot (excluding the current one).</td>
-</tr>
-<tr>
-<td>end-snapshot-id</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>When streaming is set to false, you need to specify the start-snapshot-id to read the incremental data within two intervals (snapshot1, snapshot2].</td>
-</tr>
-<tr>
-<td>other table parameters</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>All parameters of an Amoro table can be dynamically modified through SQL Hint, but only for the current task. For a list of specific parameters, please refer to <a href="../configurations/">Table Configurations</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>snapshot-id<img width=100/></td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>Reading the full data of a specific snapshot only works when streaming is set to false.</td>
+ </tr>
+ <tr>
+ <td>as-of-timestamp</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>Reading the full data of the latest snapshot taken before the specified timestamp only works when streaming is set to false.</td>
+ </tr>
+ <tr>
+ <td>start-snapshot-id</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>When streaming is set to false, you need to specify the end-snapshot-id to read the incremental data within two intervals (snapshot1, snapshot2]. When streaming is set to true, you can read the incremental data after the specified snapshot. If not specified, it will read the incremental data after the current snapshot (excluding the current one).</td>
+ </tr>
+ <tr>
+ <td>end-snapshot-id</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>When streaming is set to false, you need to specify the start-snapshot-id to read the incremental data within two intervals (snapshot1, snapshot2].</td>
+ </tr>
+ <tr>
+ <td>other table parameters</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>All parameters of an Amoro table can be dynamically modified through SQL Hint, but only for the current task. For a list of specific parameters, please refer to <a href="../configurations/">Table Configurations</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
+ </tr>
+ </tbody>
</table>
<h3 id="batch-mode-primary-key-table">Batch mode (primary key table)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Merge on Read the current mixed-format table and return append-only data.
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> keyed <span style="color:#75715e">/*+ OPTIONS('streaming'='false', 'scan.startup.mode'='earliest')*/</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> keyed <span style="color:#75715e">/*+ OPTIONS('streaming'='false', 'scan.startup.mode'='earliest')*/</span>;
</span></span></code></pre></div><h3 id="streaming-mode">Streaming mode</h3>
<p>Amoro supports reading incremental data from FileStore or LogStore in streaming mode.</p>
<h3 id="streaming-mode-logstore">Streaming mode (LogStore)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in streaming mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Enable dynamic table parameter configuration to make hint options configured in Flink SQL effective
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled<span style="color:#f92672">=</span><span style="color:#66d9ef">true</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> test_table <span style="color:#75715e">/*+ OPTIONS('mixed-format.read.mode'='log') */</span>;
</span></span></code></pre></div><p>The following Hint Options are supported:</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>mixed-format.read.mode</td>
-<td>file</td>
-<td>String</td>
-<td>No</td>
-<td>To specify the type of data to read from an Amoro table, either File or Log, use the mixed-format.read.mode parameter. If the value is set to log, the Log configuration must be enabled.</td>
-</tr>
-<tr>
-<td>scan.startup.mode<img width=90/></td>
-<td>latest</td>
-<td>String</td>
-<td>No</td>
-<td>The valid values are ’earliest’, ’latest’, ’timestamp’, ‘group-offsets’ and ‘specific-offsets’. <br> ’earliest’ reads the data from the earliest offset possible.<br> ’latest’ reads the data from the latest offset. <br>‘timestamp’ reads from a specified time position, which requires configuring the ‘scan.startup.timestamp-millis’ parameter. <br>‘group-offsets’ reads [...]
-</tr>
-<tr>
-<td>scan.startup.timestamp-millis</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>Valid when ‘scan.startup.mode’ = ’timestamp’, reads data from the specified Kafka time with a millisecond timestamp starting at 00:00:00.000 GMT on 1 Jan 1970</td>
-</tr>
-<tr>
-<td>scan.startup.specific-offsets</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>specify offsets for each partition in case of ‘specific-offsets’ startup mode, e.g. ‘partition:0,offset:42;partition:1,offset:300’.</td>
-</tr>
-<tr>
-<td>properties.group.id</td>
-<td>(none)</td>
-<td>String</td>
-<td>If the LogStore for an Amoro table is Kafka, it is mandatory to provide its details while querying the table. Otherwise, it can be left empty.</td>
-<td>The group id used to read the Kafka Topic</td>
-</tr>
-<tr>
-<td>properties.pulsar.admin.adminUrl</td>
-<td>(none)</td>
-<td>String</td>
-<td>Required if LogStore is pulsar, otherwise not required</td>
-<td>Pulsar admin’s HTTP URL， e.g. <a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a></td>
-</tr>
-<tr>
-<td>properties.*</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Parameters for Logstore: <br>For Logstore with Kafka (’log-store.type’=‘kafka’ default value), all other parameters supported by the Kafka Consumer can be set by prefixing properties. to the parameter name, for example, ‘properties.batch.size’=‘16384’. The complete parameter information can be found in the <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka official documentation</a>; <br>For LogStore set to Pulsar ( [...]
-</tr>
-<tr>
-<td>log.consumer.changelog.modes</td>
-<td>all-kinds</td>
-<td>String</td>
-<td>No</td>
-<td>The type of RowKind that will be generated when reading log data, supports: all-kinds, append-only.<br>all-kinds: will read cdc data, including +I/-D/-U/+U;<br>append-only: will only generate Insert data, recommended to use this configuration when reading without primary key.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>mixed-format.read.mode</td>
+ <td>file</td>
+ <td>String</td>
+ <td>No</td>
+ <td>To specify the type of data to read from an Amoro table, either File or Log, use the mixed-format.read.mode parameter. If the value is set to log, the Log configuration must be enabled.</td>
+ </tr>
+ <tr>
+ <td>scan.startup.mode<img width=90/></td>
+ <td>latest</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The valid values are ’earliest’, ’latest’, ’timestamp’, ‘group-offsets’ and ‘specific-offsets’. <br> ’earliest’ reads the data from the earliest offset possible.<br> ’latest’ reads the data from the latest offset. <br>‘timestamp’ reads from a specified time position, which requires configuring the ‘scan.startup.timestamp-millis’ parameter. <br>‘group-offsets&rsq [...]
+ </tr>
+ <tr>
+ <td>scan.startup.timestamp-millis</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>Valid when ‘scan.startup.mode’ = ’timestamp’, reads data from the specified Kafka time with a millisecond timestamp starting at 00:00:00.000 GMT on 1 Jan 1970</td>
+ </tr>
+ <tr>
+ <td>scan.startup.specific-offsets</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>specify offsets for each partition in case of ‘specific-offsets’ startup mode, e.g. ‘partition:0,offset:42;partition:1,offset:300’.</td>
+ </tr>
+ <tr>
+ <td>properties.group.id</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>If the LogStore for an Amoro table is Kafka, it is mandatory to provide its details while querying the table. Otherwise, it can be left empty.</td>
+ <td>The group id used to read the Kafka Topic</td>
+ </tr>
+ <tr>
+ <td>properties.pulsar.admin.adminUrl</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>Required if LogStore is pulsar, otherwise not required</td>
+ <td>Pulsar admin’s HTTP URL， e.g. <a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a></td>
+ </tr>
+ <tr>
+ <td>properties.*</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Parameters for Logstore: <br>For Logstore with Kafka (’log-store.type’=‘kafka’ default value), all other parameters supported by the Kafka Consumer can be set by prefixing properties. to the parameter name, for example, ‘properties.batch.size’=‘16384’. The complete parameter information can be found in the <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka official documentation</a>; <br>For LogStore set t [...]
+ </tr>
+ <tr>
+ <td>log.consumer.changelog.modes</td>
+ <td>all-kinds</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The type of RowKind that will be generated when reading log data, supports: all-kinds, append-only.<br>all-kinds: will read cdc data, including +I/-D/-U/+U;<br>append-only: will only generate Insert data, recommended to use this configuration when reading without primary key.</td>
+ </tr>
+ </tbody>
</table>
<blockquote>
<p><strong>Notes</strong></p>
@@ -818,121 +818,121 @@
</blockquote>
<h3 id="streaming-mode-filestore-non-primary-key-table">Streaming mode (FileStore non-primary key table)</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in streaming mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Enable dynamic table parameter configuration to make hint options configured in Flink SQL effective
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Read incremental data after the current snapshot.
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('monitor-interval'='1s')*/</span> ;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> unkeyed <span style="color:#75715e">/*+ OPTIONS('monitor-interval'='1s')*/</span> ;
</span></span></code></pre></div><p>Hint Options</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>streaming</td>
-<td>true</td>
-<td>Boolean</td>
-<td>No</td>
-<td>Reads bounded data or unbounded data in a streaming mode, false: reads bounded data, true: reads unbounded data</td>
-</tr>
-<tr>
-<td>mixed-format.read.mode</td>
-<td>file</td>
-<td>String</td>
-<td>No</td>
-<td>To specify the type of data to read from an Amoro table, either File or Log, use the mixed-format.read.mode parameter. If the value is set to log, the Log configuration must be enabled.</td>
-</tr>
-<tr>
-<td>monitor-interval<img width=120/></td>
-<td>10s</td>
-<td>Duration</td>
-<td>No</td>
-<td>The mixed-format.read.mode = file parameter needs to be set for this to take effect. The time interval for monitoring newly added data files</td>
-</tr>
-<tr>
-<td>start-snapshot-id</td>
-<td>(none)</td>
-<td>Long</td>
-<td>No</td>
-<td>To read incremental data starting from a specified snapshot (excluding the data in the start-snapshot-id snapshot), specify the snapshot ID using the start-snapshot-id parameter. If not specified, the reader will start reading from the snapshot after the current one (excluding the data in the current snapshot).</td>
-</tr>
-<tr>
-<td>other table parameters</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>streaming</td>
+ <td>true</td>
+ <td>Boolean</td>
+ <td>No</td>
+ <td>Reads bounded data or unbounded data in a streaming mode, false: reads bounded data, true: reads unbounded data</td>
+ </tr>
+ <tr>
+ <td>mixed-format.read.mode</td>
+ <td>file</td>
+ <td>String</td>
+ <td>No</td>
+ <td>To specify the type of data to read from an Amoro table, either File or Log, use the mixed-format.read.mode parameter. If the value is set to log, the Log configuration must be enabled.</td>
+ </tr>
+ <tr>
+ <td>monitor-interval<img width=120/></td>
+ <td>10s</td>
+ <td>Duration</td>
+ <td>No</td>
+ <td>The mixed-format.read.mode = file parameter needs to be set for this to take effect. The time interval for monitoring newly added data files</td>
+ </tr>
+ <tr>
+ <td>start-snapshot-id</td>
+ <td>(none)</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>To read incremental data starting from a specified snapshot (excluding the data in the start-snapshot-id snapshot), specify the snapshot ID using the start-snapshot-id parameter. If not specified, the reader will start reading from the snapshot after the current one (excluding the data in the current snapshot).</td>
+ </tr>
+ <tr>
+ <td>other table parameters</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
+ </tr>
+ </tbody>
</table>
<h3 id="streaming-mode-filestore-primary-key-table">Streaming Mode (FileStore primary key table)</h3>
<p>After using CDC (Change Data Capture) to ingest data into the lake, you can use the Flink engine to read both stock data and incremental data in the same task without restarting the task, and ensure consistent data reading. Amoro Source will save the file offset information in the Flink state.</p>
<p>In this way, the task can continue to read data from the last read offset position, ensuring data consistency and being able to process newly arrived incremental data.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- Run Flink tasks in streaming mode in the current session
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> execution.runtime<span style="color:#f92672">-</span><span style="color:#66d9ef">mode</span> <span style="color:#f92672">=</span> streaming;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Enable dynamic table parameter configuration to make hint options configured in Flink SQL effective
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SET</span> <span style="color:#66d9ef">table</span>.<span style="color:#66d9ef">dynamic</span><span style="color:#f92672">-</span><span style="color:#66d9ef">table</span><span style="color:#f92672">-</span><span style="color:#66d9ef">options</span>.enabled <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Incremental unified reading of BaseStore and ChangeStore
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> keyed <span style="color:#75715e">/*+ OPTIONS('streaming'='true', 'scan.startup.mode'='earliest')*/</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> keyed <span style="color:#75715e">/*+ OPTIONS('streaming'='true', 'scan.startup.mode'='earliest')*/</span>;
</span></span></code></pre></div><p>Hint Options</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>streaming</td>
-<td>true</td>
-<td>String</td>
-<td>No</td>
-<td>Reads bounded data or unbounded data in a streaming mode, false: reads bounded data, true: reads unbounded data</td>
-</tr>
-<tr>
-<td>mixed-format.read.mode</td>
-<td>file</td>
-<td>String</td>
-<td>No</td>
-<td>Specifies the data to read from an Amoro table, either file or log. If the value is “log”, Log configuration must be enabled</td>
-</tr>
-<tr>
-<td>monitor-interval</td>
-<td>10s</td>
-<td>String</td>
-<td>No</td>
-<td>This parameter only takes effect when mixed-format.read.mode = file. It sets the time interval for monitoring newly added data files</td>
-</tr>
-<tr>
-<td>scan.startup.mode</td>
-<td>latest</td>
-<td>String</td>
-<td>No</td>
-<td>The valid values are ’earliest’, ’latest’. ’earliest’ reads the full table data and will continue to read incremental data when streaming=true. ’latest’ reads only the data after the current snapshot, not including the data in the current snapshot.</td>
-</tr>
-<tr>
-<td>other table parameters</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>streaming</td>
+ <td>true</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Reads bounded data or unbounded data in a streaming mode, false: reads bounded data, true: reads unbounded data</td>
+ </tr>
+ <tr>
+ <td>mixed-format.read.mode</td>
+ <td>file</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Specifies the data to read from an Amoro table, either file or log. If the value is “log”, Log configuration must be enabled</td>
+ </tr>
+ <tr>
+ <td>monitor-interval</td>
+ <td>10s</td>
+ <td>String</td>
+ <td>No</td>
+ <td>This parameter only takes effect when mixed-format.read.mode = file. It sets the time interval for monitoring newly added data files</td>
+ </tr>
+ <tr>
+ <td>scan.startup.mode</td>
+ <td>latest</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The valid values are ’earliest’, ’latest’. ’earliest’ reads the full table data and will continue to read incremental data when streaming=true. ’latest’ reads only the data after the current snapshot, not including the data in the current snapshot.</td>
+ </tr>
+ <tr>
+ <td>other table parameters</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
+ </tr>
+ </tbody>
</table>
<h2 id="writing-with-sql">Writing With SQL</h2>
<p>Amoro tables support writing data to LogStore or FileStore using Flink SQL.</p>
@@ -941,7 +941,7 @@
<p>INSERT OVERWRITE only allows running in Flink Batch mode.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">INSERT</span> OVERWRITE unkeyed <span style="color:#66d9ef">VALUES</span> (<span style="color:#ae81ff">1</span>, <span style="color:#e6db74">'a'</span>, <span style="color:#e6db74">'2022-07-01'</span>);
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- It is also possible to overwrite data for a specific partition:
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
+</span></span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">INSERT</span> OVERWRITE <span style="color:#f92672">`</span>mixed_catalog<span style="color:#f92672">`</span>.<span style="color:#f92672">`</span>mixed_db<span style="color:#f92672">`</span>.<span style="color:#f92672">`</span>unkeyed<span style="color:#f92672">`</span> PARTITION(<span style="color:#66d9ef">data</span><span style="color:#f92672">=</span><span style="color:#e6db74">'2022-07-01'</span>) <sp [...]
</span></span></code></pre></div><p>For non-partitioned tables, INSERT OVERWRITE will overwrite the entire data in the table.</p>
<h3 id="insert-into">INSERT INTO</h3>
@@ -952,86 +952,86 @@
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> id, name <span style="color:#66d9ef">from</span> <span style="color:#f92672">`</span><span style="color:#66d9ef">source</span><span style="color:#f92672">`</span>;
</span></span></code></pre></div><p>Hint Options</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>mixed-format.emit.mode</td>
-<td>auto</td>
-<td>String</td>
-<td>No</td>
-<td>Data writing modes currently supported are: file, log, and auto. For example: ‘file’ means data is only written to the Filestore. ’log’ means data is only written to the Logstore. ‘file,log’ means data is written to both the Filestore and the Logstore. ‘auto’ means data is written only to the Filestore if the Logstore for the Amoro table is disabled. If the Logstore for the Amoro table is enabled, it means data is written to both the Fi [...]
-</tr>
-<tr>
-<td>mixed-format.emit.auto-write-to-logstore.watermark-gap</td>
-<td>(none)</td>
-<td>Duration</td>
-<td>No</td>
-<td>This feature is only enabled when ‘mixed-format.emit.mode’=‘auto’. If the watermark of the Amoro writers is greater than the current system timestamp minus a specific value, the writers will also write data to the Logstore. The default setting is to enable the Logstore writer immediately after the job starts. The value for this feature must be greater than 0.</td>
-</tr>
-<tr>
-<td>log.version</td>
-<td>v1</td>
-<td>String</td>
-<td>No</td>
-<td>The log data format currently has only one version, so it can be left empty</td>
-</tr>
-<tr>
-<td>sink.parallelism</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>The parallelism for writing to the Filestore and Logstore is determined separately. The parallelism for submitting the file operator is always 1.</td>
-</tr>
-<tr>
-<td>write.distribution-mode</td>
-<td>hash</td>
-<td>String</td>
-<td>No</td>
-<td>The distribution modes for writing to the Amoro table include: none and hash.</td>
-</tr>
-<tr>
-<td>write.distribution.hash-mode</td>
-<td>auto</td>
-<td>String</td>
-<td>No</td>
-<td>The hash strategy for writing to an Amoro table only takes effect when write.distribution-mode=hash. The available options are: primary-key, partition-key, primary-partition-key, and auto. primary-key: Shuffle by primary key partition-key: Shuffle by partition key primary-partition-key: Shuffle by primary key and partition key auto: If the table has both a primary key and partitions, use primary-partition-key; if the table has a primary key but no partitions, use primary-key; if the [...]
-</tr>
-<tr>
-<td>properties.pulsar.admin.adminUrl</td>
-<td>(none)</td>
-<td>String</td>
-<td>If the LogStore is Pulsar and it is required for querying, it must be filled in, otherwise it can be left empty.<img width=100/></td>
-<td>The HTTP URL for Pulsar Admin is in the format: <a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a>.</td>
-</tr>
-<tr>
-<td>properties.*</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>Parameters for Logstore: For Logstore with Kafka (’log-store.type’=‘kafka’ default value), all other parameters supported by the Kafka Consumer can be set by prefixing properties. to the parameter name, for example, ‘properties.batch.size’=‘16384’. The complete parameter information can be found in the <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka official documentation</a>; For LogStore set to Pulsar (’l [...]
-</tr>
-<tr>
-<td>other table parameters</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>mixed-format.emit.mode</td>
+ <td>auto</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Data writing modes currently supported are: file, log, and auto. For example: ‘file’ means data is only written to the Filestore. ’log’ means data is only written to the Logstore. ‘file,log’ means data is written to both the Filestore and the Logstore. ‘auto’ means data is written only to the Filestore if the Logstore for the Amoro table is disabled. If the Logstore for the Amoro table is enabled, it means data is written to b [...]
+ </tr>
+ <tr>
+ <td>mixed-format.emit.auto-write-to-logstore.watermark-gap</td>
+ <td>(none)</td>
+ <td>Duration</td>
+ <td>No</td>
+ <td>This feature is only enabled when ‘mixed-format.emit.mode’=‘auto’. If the watermark of the Amoro writers is greater than the current system timestamp minus a specific value, the writers will also write data to the Logstore. The default setting is to enable the Logstore writer immediately after the job starts. The value for this feature must be greater than 0.</td>
+ </tr>
+ <tr>
+ <td>log.version</td>
+ <td>v1</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The log data format currently has only one version, so it can be left empty</td>
+ </tr>
+ <tr>
+ <td>sink.parallelism</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The parallelism for writing to the Filestore and Logstore is determined separately. The parallelism for submitting the file operator is always 1.</td>
+ </tr>
+ <tr>
+ <td>write.distribution-mode</td>
+ <td>hash</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The distribution modes for writing to the Amoro table include: none and hash.</td>
+ </tr>
+ <tr>
+ <td>write.distribution.hash-mode</td>
+ <td>auto</td>
+ <td>String</td>
+ <td>No</td>
+ <td>The hash strategy for writing to an Amoro table only takes effect when write.distribution-mode=hash. The available options are: primary-key, partition-key, primary-partition-key, and auto. primary-key: Shuffle by primary key partition-key: Shuffle by partition key primary-partition-key: Shuffle by primary key and partition key auto: If the table has both a primary key and partitions, use primary-partition-key; if the table has a primary key but no partitions, use primary-ke [...]
+ </tr>
+ <tr>
+ <td>properties.pulsar.admin.adminUrl</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>If the LogStore is Pulsar and it is required for querying, it must be filled in, otherwise it can be left empty.<img width=100/></td>
+ <td>The HTTP URL for Pulsar Admin is in the format: <a href="http://my-broker.example.com:8080">http://my-broker.example.com:8080</a>.</td>
+ </tr>
+ <tr>
+ <td>properties.*</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>Parameters for Logstore: For Logstore with Kafka (’log-store.type’=‘kafka’ default value), all other parameters supported by the Kafka Consumer can be set by prefixing properties. to the parameter name, for example, ‘properties.batch.size’=‘16384’. The complete parameter information can be found in the <a href="https://kafka.apache.org/documentation/#consumerconfigs">Kafka official documentation</a>; For LogStore set to Pulsar [...]
+ </tr>
+ <tr>
+ <td>other table parameters</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
+ </tr>
+ </tbody>
</table>
<h2 id="lookup-join-with-sql">Lookup join with SQL</h2>
<p>A Lookup Join is used to enrich a table with data that is queried from Amoro Table. The join requires one table to have a processing time attribute and the other table to be backed by a lookup source connector.</p>
<p>The following example shows the syntax to specify a lookup join.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- amoro flink connector and can be used for lookup joins
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TEMPORARY</span> <span style="color:#66d9ef">TABLE</span> Customers (
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TEMPORARY</span> <span style="color:#66d9ef">TABLE</span> Customers (
</span></span><span style="display:flex;"><span> id INT,
</span></span><span style="display:flex;"><span> name STRING,
</span></span><span style="display:flex;"><span> country STRING,
@@ -1046,7 +1046,7 @@
</span></span><span style="display:flex;"><span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- Create a temporary left table, like from kafka
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TEMPORARY</span> <span style="color:#66d9ef">TABLE</span> orders (
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TEMPORARY</span> <span style="color:#66d9ef">TABLE</span> orders (
</span></span><span style="display:flex;"><span> order_id INT,
</span></span><span style="display:flex;"><span> total INT,
</span></span><span style="display:flex;"><span> customer_id INT,
@@ -1060,79 +1060,79 @@
</span></span><span style="display:flex;"><span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- enrich each order with customer information
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">SELECT</span> o.order_id, o.total, <span style="color:#66d9ef">c</span>.country, <span style="color:#66d9ef">c</span>.zip
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> o.order_id, o.total, <span style="color:#66d9ef">c</span>.country, <span style="color:#66d9ef">c</span>.zip
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> Orders <span style="color:#66d9ef">AS</span> o
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">JOIN</span> Customers <span style="color:#66d9ef">FOR</span> SYSTEM_TIME <span style="color:#66d9ef">AS</span> <span style="color:#66d9ef">OF</span> o.proc_time <span style="color:#66d9ef">AS</span> <span style="color:#66d9ef">c</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ON</span> o.customer_id <span style="color:#f92672">=</span> <span style="color:#66d9ef">c</span>.id;
</span></span></code></pre></div><p>Lookup Options</p>
<table>
-<thead>
-<tr>
-<th>Key</th>
-<th>Default Value</th>
-<th>Type</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>lookup.cache.max-rows</td>
-<td>10000</td>
-<td>Long</td>
-<td>No</td>
-<td>The maximum number of rows in the lookup cache, beyond which the oldest row will expire.</td>
-</tr>
-<tr>
-<td>lookup.reloading.interval</td>
-<td>10s</td>
-<td>Duration</td>
-<td>No</td>
-<td>Configuration option for specifying the interval in seconds to reload lookup data in RocksDB.</td>
-</tr>
-<tr>
-<td>lookup.cache.ttl-after-write</td>
-<td>0s</td>
-<td>Duration</td>
-<td>No</td>
-<td>The TTL after which the row will expire in the lookup cache.</td>
-</tr>
-<tr>
-<td>rocksdb.auto-compactions</td>
-<td>false</td>
-<td>Boolean</td>
-<td>No</td>
-<td>Enable automatic compactions during the initialization process. After the initialization completed, will enable the auto_compaction.</td>
-</tr>
-<tr>
-<td>rocksdb.writing-threads</td>
-<td>5</td>
-<td>Int</td>
-<td>No</td>
-<td>Writing data into rocksDB thread number.</td>
-</tr>
-<tr>
-<td>rocksdb.block-cache.capacity</td>
-<td>1048576</td>
-<td>Long</td>
-<td>No</td>
-<td>Use the LRUCache strategy for blocks, the size of the BlockCache can be configured based on your memory requirements and available system resources.</td>
-</tr>
-<tr>
-<td>rocksdb.block-cache.numShardBits</td>
-<td>-1</td>
-<td>Int</td>
-<td>No</td>
-<td>Use the LRUCache strategy for blocks. The cache is sharded to 2^numShardBits shards, by hash of the key. Default is -1, means it is automatically determined: every shard will be at least 512KB and number of shard bits will not exceed 6.</td>
-</tr>
-<tr>
-<td>other table parameters</td>
-<td>(none)</td>
-<td>String</td>
-<td>No</td>
-<td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Key</th>
+ <th>Default Value</th>
+ <th>Type</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>lookup.cache.max-rows</td>
+ <td>10000</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>The maximum number of rows in the lookup cache, beyond which the oldest row will expire.</td>
+ </tr>
+ <tr>
+ <td>lookup.reloading.interval</td>
+ <td>10s</td>
+ <td>Duration</td>
+ <td>No</td>
+ <td>Configuration option for specifying the interval in seconds to reload lookup data in RocksDB.</td>
+ </tr>
+ <tr>
+ <td>lookup.cache.ttl-after-write</td>
+ <td>0s</td>
+ <td>Duration</td>
+ <td>No</td>
+ <td>The TTL after which the row will expire in the lookup cache.</td>
+ </tr>
+ <tr>
+ <td>rocksdb.auto-compactions</td>
+ <td>false</td>
+ <td>Boolean</td>
+ <td>No</td>
+ <td>Enable automatic compactions during the initialization process. After the initialization completed, will enable the auto_compaction.</td>
+ </tr>
+ <tr>
+ <td>rocksdb.writing-threads</td>
+ <td>5</td>
+ <td>Int</td>
+ <td>No</td>
+ <td>Writing data into rocksDB thread number.</td>
+ </tr>
+ <tr>
+ <td>rocksdb.block-cache.capacity</td>
+ <td>1048576</td>
+ <td>Long</td>
+ <td>No</td>
+ <td>Use the LRUCache strategy for blocks, the size of the BlockCache can be configured based on your memory requirements and available system resources.</td>
+ </tr>
+ <tr>
+ <td>rocksdb.block-cache.numShardBits</td>
+ <td>-1</td>
+ <td>Int</td>
+ <td>No</td>
+ <td>Use the LRUCache strategy for blocks. The cache is sharded to 2^numShardBits shards, by hash of the key. Default is -1, means it is automatically determined: every shard will be at least 512KB and number of shard bits will not exceed 6.</td>
+ </tr>
+ <tr>
+ <td>other table parameters</td>
+ <td>(none)</td>
+ <td>String</td>
+ <td>No</td>
+ <td>All parameters of an Amoro table can be dynamically modified through SQL Hints, but they only take effect for this specific task. For the specific parameter list, please refer to the <a href="../configurations/">Table Configuration</a>. For permissions-related configurations on the catalog, they can also be configured in Hint using parameters such as <a href="../flink-ddl/#flink-sql">properties.auth.XXX in catalog DDL</a></td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/flink-using-logstore/index.html b/output/docs/latest/flink-using-logstore/index.html
index e999445a3..631d87e7c 100644
--- a/output/docs/latest/flink-using-logstore/index.html
+++ b/output/docs/latest/flink-using-logstore/index.html
@@ -640,49 +640,49 @@
<p>Users can enable LogStore by configuring the following parameters when creating an Amoro table. For specific configurations, please refer to <a href="../configurations/#logstore-configurations">LogStore related configurations</a>.</p>
<h2 id="overview">Overview</h2>
<table>
-<thead>
-<tr>
-<th>Flink</th>
-<th>Kafka</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Flink 1.15</td>
-<td>✔</td>
-</tr>
-<tr>
-<td>Flink 1.16</td>
-<td>✔</td>
-</tr>
-<tr>
-<td>Flink 1.17</td>
-<td>✔</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Flink</th>
+ <th>Kafka</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Flink 1.15</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Flink 1.16</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Flink 1.17</td>
+ <td>✔</td>
+ </tr>
+ </tbody>
</table>
<p>Kafka as LogStore Version Description:</p>
<table>
-<thead>
-<tr>
-<th>Flink Version</th>
-<th>Kafka Versions</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>1.15.x</td>
-<td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
-</tr>
-<tr>
-<td>1.16.x</td>
-<td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
-</tr>
-<tr>
-<td>1.17.x</td>
-<td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Flink Version</th>
+ <th>Kafka Versions</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>1.15.x</td>
+ <td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
+ </tr>
+ <tr>
+ <td>1.16.x</td>
+ <td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
+ </tr>
+ <tr>
+ <td>1.17.x</td>
+ <td>0.10.2.*<br> 0.11.*<br> 1.*<br> 2.*<br> 3.*</td>
+ </tr>
+ </tbody>
</table>
<h3 id="prerequisites-for-using-logstore">Prerequisites for using LogStore</h3>
<p>When creating an Amoro table, LogStore needs to be enabled.</p>
@@ -704,7 +704,7 @@
<li>You can also use Flink SQL to create tables in Flink-SQL-Client</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- First use the use catalog command to switch to the mixed-format catalog.
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TABLE</span> db.log_table (
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">CREATE</span> <span style="color:#66d9ef">TABLE</span> db.log_table (
</span></span><span style="display:flex;"><span> id int,
</span></span><span style="display:flex;"><span> name string,
</span></span><span style="display:flex;"><span> ts <span style="color:#66d9ef">timestamp</span>,
diff --git a/output/docs/latest/flink/cdc-ingestion/index.html b/output/docs/latest/flink/cdc-ingestion/index.html
index 961ffdbd1..79a3ff2e3 100644
--- a/output/docs/latest/flink/cdc-ingestion/index.html
+++ b/output/docs/latest/flink/cdc-ingestion/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-cdc-ingestion/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-cdc-ingestion/">
</head>
diff --git a/output/docs/latest/flink/datastream/index.html b/output/docs/latest/flink/datastream/index.html
index d34e16c23..9cb33e153 100644
--- a/output/docs/latest/flink/datastream/index.html
+++ b/output/docs/latest/flink/datastream/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-datastream/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-datastream/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-datastream/">
</head>
diff --git a/output/docs/latest/flink/ddl/index.html b/output/docs/latest/flink/ddl/index.html
index 00839e3b8..f242a4f66 100644
--- a/output/docs/latest/flink/ddl/index.html
+++ b/output/docs/latest/flink/ddl/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-ddl/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-ddl/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-ddl/">
</head>
diff --git a/output/docs/latest/flink/dml/index.html b/output/docs/latest/flink/dml/index.html
index db028da5a..8461c163d 100644
--- a/output/docs/latest/flink/dml/index.html
+++ b/output/docs/latest/flink/dml/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-dml/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-dml/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-dml/">
</head>
diff --git a/output/docs/latest/flink/getting-started/index.html b/output/docs/latest/flink/getting-started/index.html
index af2ae1798..42e6aba8b 100644
--- a/output/docs/latest/flink/getting-started/index.html
+++ b/output/docs/latest/flink/getting-started/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-getting-started/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-getting-started/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-getting-started/">
</head>
diff --git a/output/docs/latest/flink/using-logstore/index.html b/output/docs/latest/flink/using-logstore/index.html
index 04148e774..2c9d8eb39 100644
--- a/output/docs/latest/flink/using-logstore/index.html
+++ b/output/docs/latest/flink/using-logstore/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/flink-using-logstore/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/flink-using-logstore/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/flink-using-logstore/">
</head>
diff --git a/output/docs/latest/formats/iceberg/index.html b/output/docs/latest/formats/iceberg/index.html
index 60c8bc96d..b095ffcd2 100644
--- a/output/docs/latest/formats/iceberg/index.html
+++ b/output/docs/latest/formats/iceberg/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/iceberg-format/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/iceberg-format/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/iceberg-format/">
</head>
diff --git a/output/docs/latest/formats/index.xml b/output/docs/latest/formats/index.xml
index 6b067fc45..2e1d5d4ae 100644
--- a/output/docs/latest/formats/index.xml
+++ b/output/docs/latest/formats/index.xml
@@ -12,35 +12,35 @@
<link>https://amoro.apache.org/docs/latest/iceberg-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/iceberg-format/</guid>
- <description>Iceberg Format Iceberg format refers to Apache Iceberg table, which is an open table format for large analytical datasets designed to provide scalable, efficient, and secure data storage and query solutions. It supports data operations on multiple storage backends and provides features such as ACID transactions, multi-version control, and schema evolution, making data management and querying more flexible and convenient.
With the release of Iceberg v2, Iceberg addr [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Mixed-Hive</title>
<link>https://amoro.apache.org/docs/latest/mixed-hive-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/mixed-hive-format/</guid>
- <description>Mixed-Hive Format Mixed-Hive format is a format that has better compatibility with Hive than Mixed-Iceberg format. Mixed-Hive format uses a Hive table as the BaseStore and an Iceberg table as the ChangeStore. Mixed-Hive format supports:
schema, partition, and types consistent with Hive format Using the Hive connector to read and write Mixed-Hive format tables as Hive tables Upgrading a Hive table in-place to a Mixed-Hive format table without data rewriting or migra [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Mixed-Iceberg</title>
<link>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</guid>
- <description>Mixed-Iceberg Format Compared with Iceberg format, Mixed-Iceberg format provides more features:
Stronger primary key constraints that also apply to Spark OLAP performance that is production-ready for real-time data warehouses through the auto-bucket mechanism LogStore configuration that can reduce data pipeline latency from minutes to milliseconds/seconds Transaction conflict resolution mechanism that enables concurrent writes with the same primary key The design i [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Overview</title>
<link>https://amoro.apache.org/docs/latest/formats-overview/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/formats-overview/</guid>
- <description>Formats Overview Table format (aka. format) was first proposed by Iceberg, which can be described as follows:
It defines the relationship between tables and files, and any engine can query and retrieve data files according to the table format. New formats such as Iceberg/Delta/Hudi further define the relationship between tables and snapshots, and the relationship between snapshots and files. All write operations on the table will generate new snapshots, and all rea [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Paimon</title>
<link>https://amoro.apache.org/docs/latest/paimon-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/paimon-format/</guid>
- <description>Paimon Format Paimon format refers to Apache Paimon table. Paimon is a streaming data lake platform with high-speed data ingestion, changelog tracking and efficient real-time analytics.
By registering Paimon&rsquo;s catalog with Amoro, users can view information such as Schema, Options, Files, Snapshots, DDLs, Compaction information, and more for Paimon tables. Furthermore, they can operate on Paimon tables using Spark SQL in the Terminal. The current supported [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/formats/mixed-hive/index.html b/output/docs/latest/formats/mixed-hive/index.html
index 1b9b997ba..0323a0fa3 100644
--- a/output/docs/latest/formats/mixed-hive/index.html
+++ b/output/docs/latest/formats/mixed-hive/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/mixed-hive-format/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/mixed-hive-format/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/mixed-hive-format/">
</head>
diff --git a/output/docs/latest/formats/mixed-iceberg/index.html b/output/docs/latest/formats/mixed-iceberg/index.html
index f2509248f..d2cfc6dd1 100644
--- a/output/docs/latest/formats/mixed-iceberg/index.html
+++ b/output/docs/latest/formats/mixed-iceberg/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/mixed-iceberg-format/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/mixed-iceberg-format/">
</head>
diff --git a/output/docs/latest/formats/overview/index.html b/output/docs/latest/formats/overview/index.html
index ae6c21053..cd04b2f69 100644
--- a/output/docs/latest/formats/overview/index.html
+++ b/output/docs/latest/formats/overview/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/formats-overview/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/formats-overview/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/formats-overview/">
</head>
diff --git a/output/docs/latest/formats/paimon/index.html b/output/docs/latest/formats/paimon/index.html
index 429dcb173..f3d7e248b 100644
--- a/output/docs/latest/formats/paimon/index.html
+++ b/output/docs/latest/formats/paimon/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/paimon-format/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/paimon-format/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/paimon-format/">
</head>
diff --git a/output/docs/latest/index.html b/output/docs/latest/index.html
index 06adb255a..d3f525aa4 100644
--- a/output/docs/latest/index.html
+++ b/output/docs/latest/index.html
@@ -32,7 +32,7 @@
<!DOCTYPE html>
<html>
<head>
- <meta name="generator" content="Hugo 0.130.0">
+ <meta name="generator" content="Hugo 0.154.2">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
@@ -673,65 +673,65 @@ For details, please refer to: <a href="https://paimon.apache.org/docs/master/">P
<h3 id="mixed-format">Mixed format</h3>
<p>Amoro support multiple processing engines for Mixed format as below:</p>
<table>
-<thead>
-<tr>
-<th>Processing Engine</th>
-<th>Version</th>
-<th>Batch Read</th>
-<th>Batch Write</th>
-<th>Batch Overwrite</th>
-<th>Streaming Read</th>
-<th>Streaming Write</th>
-<th>Create Table</th>
-<th>Alter Table</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Flink</td>
-<td>1.16.x, 1.17.x, 1.18.x</td>
-<td>✔</td>
-<td>✔</td>
-<td>✖</td>
-<td>✔</td>
-<td>✔</td>
-<td>✔</td>
-<td>✖</td>
-</tr>
-<tr>
-<td>Spark</td>
-<td>3.2, 3.3, 3.5</td>
-<td>✔</td>
-<td>✔</td>
-<td>✔</td>
-<td>✖</td>
-<td>✖</td>
-<td>✔</td>
-<td>✔</td>
-</tr>
-<tr>
-<td>Hive</td>
-<td>2.x, 3.x</td>
-<td>✔</td>
-<td>✖</td>
-<td>✔</td>
-<td>✖</td>
-<td>✖</td>
-<td>✖</td>
-<td>✔</td>
-</tr>
-<tr>
-<td>Trino</td>
-<td>406</td>
-<td>✔</td>
-<td>✖</td>
-<td>✔</td>
-<td>✖</td>
-<td>✖</td>
-<td>✖</td>
-<td>✔</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Processing Engine</th>
+ <th>Version</th>
+ <th>Batch Read</th>
+ <th>Batch Write</th>
+ <th>Batch Overwrite</th>
+ <th>Streaming Read</th>
+ <th>Streaming Write</th>
+ <th>Create Table</th>
+ <th>Alter Table</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Flink</td>
+ <td>1.16.x, 1.17.x, 1.18.x</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ </tr>
+ <tr>
+ <td>Spark</td>
+ <td>3.2, 3.3, 3.5</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Hive</td>
+ <td>2.x, 3.x</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ <tr>
+ <td>Trino</td>
+ <td>406</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✔</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✖</td>
+ <td>✔</td>
+ </tr>
+ </tbody>
</table>
<h2 id="user-cases">User cases</h2>
<h3 id="self-managed-streaming-lakehouse">Self-managed streaming Lakehouse</h3>
diff --git a/output/docs/latest/index.xml b/output/docs/latest/index.xml
index 79b0847a1..68f88434f 100644
--- a/output/docs/latest/index.xml
+++ b/output/docs/latest/index.xml
@@ -12,231 +12,231 @@
<link>https://amoro.apache.org/docs/latest/readme/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/readme/</guid>
- <description>Amoro Docs This directory contains the documentation content of Amoro.
The documentation is written in Markdown format. The images referenced in the documentation are saved in the images folder. Style Proper nouns should start with a capital letter, like Hadoop、Hive、Iceberg、Amoro </description>
+ <description><h2 id="amoro-docs">Amoro Docs</h2>
<p>This directory contains the documentation content of Amoro.</p>
<ul>
<li>The documentation is written in Markdown format.</li>
<li>The images referenced in the documentation are saved in the <code>images</code> folder.</li>
</ul>
<h3 id="style">Style</h3>
<ul>
<li>Proper nouns should start [...]
</item>
<item>
<title>AMS Configuration</title>
<link>https://amoro.apache.org/docs/latest/ams-config/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/ams-config/</guid>
- <description>AMS Configuration Amoro Management Service Configuration The configuration options for Amoro Management Service (AMS).
Key Default Description admin-password admin The administrator password admin-username admin The administrator account name. auto-create-tags.enabled true Enable creating tags. auto-create-tags.interval 1 min Interval for creating tags. auto-create-tags.thread-count 3 The number of threads used for creating tags. blocker.timeout 1 min Session timeo [...]
+ <description><!--
 ~ Licensed to the Apache Software Foundation (ASF) under one
 ~ or more contributor license agreements. See the NOTICE file
 ~ distributed with this work for additional information
 ~ regarding copyright ownership. The ASF licenses this file
 ~ to you under the Apache License, Version 2.0 (the
 ~ "License"); you may not use this file except in compliance
 ~ with the License. You may obtain a copy of the License at& [...]
</item>
<item>
<title>Catalogs</title>
<link>https://amoro.apache.org/docs/latest/catalogs/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/catalogs/</guid>
- <description>Catalogs Introduce multi-catalog A catalog is a metadata namespace that stores information about databases, tables, views, indexes, users, and UDFs. It provides a higher-level namespace for table and database. Typically, a catalog is associated with a specific type of data source or cluster. In Flink, Spark and Trino, the multi-catalog feature can be used to support SQL across data sources, such as:
SELECT c.ID, c.NAME, c.AGE, o.AMOUNT FROM ${CATALOG_A}.ONLINE.CUST [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>CDC Ingestion</title>
<link>https://amoro.apache.org/docs/latest/cdc-ingestion/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/cdc-ingestion/</guid>
- <description>CDC Ingestion CDC stands for Change Data Capture, which is a broad concept, as long as it can capture the change data, it can be called CDC. Flink CDC is a Log message-based data capture tool, all the inventory and incremental data can be captured. Taking MySQL as an example, it can easily capture Binlog data through Debezium、Flink CDC and process the calculations in real time to send them to the data lake.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Configurations</title>
<link>https://amoro.apache.org/docs/latest/configurations/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/configurations/</guid>
- <description>Table Configurations Multi-level configuration management Amoro provides configurations that can be configured at the Catalog, Table, and Engine levels. The configuration priority is given first to the Engine, followed by the Table, and finally by the Catalog.
Catalog: Generally, we recommend users to set default values for tables through the Catalog properties configuration, such as Self-optimizing related configurations. Table: We also recommend users to specify [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Deployment</title>
<link>https://amoro.apache.org/docs/latest/deployment/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/deployment/</guid>
- <description>Deployment You can choose to download the stable release package from download page, or the source code form Github and compile it according to the README.
System requirements Java 8 is required. Optional: A RDBMS (PostgreSQL 14.x or higher, MySQL 5.5 or higher) Optional: ZooKeeper 3.4.x or higher Download the distribution All released package can be downloaded from download page. You can download apache-amoro-x.y.z-bin.tar.gz (x.y.z is the release number), and you [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Deployment On Kubernetes</title>
<link>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/deployment-on-kubernetes/</guid>
- <description>Deploy AMS On Kubernetes Requirements If you want to deploy AMS on Kubernetes, you’d better get a sense of the following things.
Use AMS official docker image or build AMS docker image An active Kubernetes cluster Kubectl Helm3+ Amoro Official Docker Image You can find the official docker image at Amoro Docker Hub.
The following are images that can be used in a production environment.
apache/amoro
This is an image built based on the Amoro binary distrib [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink CDC Ingestion</title>
<link>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-cdc-ingestion/</guid>
- <description>Apache CDC Ingestion CDC stands for Change Data Capture, which is a broad concept, as long as it can capture the change data, it can be called CDC. Flink CDC is a Log message-based data capture tool, all the inventory and incremental data can be captured. Taking MySQL as an example, it can easily capture Binlog data through Debezium and process the calculations in real time to send them to the data lake.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DataStream</title>
<link>https://amoro.apache.org/docs/latest/flink-datastream/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-datastream/</guid>
- <description>Flink DataStream Add maven dependency To add a dependency on Mixed-format flink connector in Maven, add the following to your pom.xml:
&lt;dependencies&gt; ... &lt;dependency&gt; &lt;groupId&gt;org.apache.amoro&lt;/groupId&gt; &lt;!-- For example: amoro-format-mixed-flink-runtime-1.15 --&gt; &lt;artifactId&gt;amoro-format-mixed-flink-runtime-${flink.minor-version}&lt;/artifactId&gt; &lt;!-- For example: 0. [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DDL</title>
<link>https://amoro.apache.org/docs/latest/flink-ddl/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-ddl/</guid>
- <description>Flink DDL Create catalogs Flink SQL The following statement can be executed to create a Flink catalog:
CREATE CATALOG &lt;catalog_name&gt; WITH ( &#39;type&#39;=&#39;mixed_iceberg&#39;, `&lt;config_key&gt;`=`&lt;config_value&gt;` ); Where &lt;catalog_name&gt; is the user-defined name of the Flink catalog, and &lt;config_key&gt;=&lt;config_value&gt; has the following configurations:
Key Default Valu [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink DML</title>
<link>https://amoro.apache.org/docs/latest/flink-dml/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-dml/</guid>
- <description>Flink DML Querying with SQL Amoro tables support reading data in stream or batch mode through Flink SQL. You can switch modes using the following methods:
-- Run Flink tasks in streaming mode in the current session SET execution.runtime-mode = streaming; -- Run Flink tasks in batch mode in the current session SET execution.runtime-mode = batch; The following Hint Options are supported:
Key Default Value Type Required Description source.parallelism (none) Intege [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Flink Getting Started</title>
<link>https://amoro.apache.org/docs/latest/flink-getting-started/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-getting-started/</guid>
- <description>Flink Getting Started Iceberg format The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at Iceberg Flink user manual for more information.
Paimon format The Paimon Format can be accessed using the Connector provided by Paimon. Refer to the documentation at Paimon Flink user manual for more information.
Mixed format The Apache Flink engine can process Amoro table data in batch and streaming mode.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Iceberg</title>
<link>https://amoro.apache.org/docs/latest/iceberg-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/iceberg-format/</guid>
- <description>Iceberg Format Iceberg format refers to Apache Iceberg table, which is an open table format for large analytical datasets designed to provide scalable, efficient, and secure data storage and query solutions. It supports data operations on multiple storage backends and provides features such as ACID transactions, multi-version control, and schema evolution, making data management and querying more flexible and convenient.
With the release of Iceberg v2, Iceberg addr [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Managing Catalogs</title>
<link>https://amoro.apache.org/docs/latest/managing-catalogs/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/managing-catalogs/</guid>
- <description>Managing Catalogs Users can import your test or online clusters through the catalog management function provided by the AMS Dashboard. Before adding a new Catalog, please read the following guidelines and select the appropriate creation according to your actual needs.
Create catalog In Amoro, the catalog is a namespace for a group of libraries and tables. Under the catalog, it is further divided into different databases, and under each database, there are different [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Managing Optimizers</title>
<link>https://amoro.apache.org/docs/latest/managing-optimizers/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/managing-optimizers/</guid>
- <description>Managing Optimizers The optimizer is the execution unit for performing self-optimizing tasks on a table. To isolate optimizing tasks on different tables and support the deployment of optimizers in different environments, Amoro has proposed the concepts of optimizer containers and optimizer groups:
Optimizer container: Encapsulate the deployment method of optimizers, there are four implementations for now: flink container based on Flink streaming job, spark containe [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Metrics</title>
<link>https://amoro.apache.org/docs/latest/metrics/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/metrics/</guid>
- <description>Metrics Amoro build a metrics system to measure the behaviours of table management processes, like how long has it been since a table last performed self-optimizing process, and how much resources does a optimizer group currently has?
There are two types of metrics provided in the Amoro metric system: Gauge and Counter.
Gauge: Provides a value of any type at a point in time. Counter: Used to count values by incrementing and decrementing.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Mixed-Hive</title>
<link>https://amoro.apache.org/docs/latest/mixed-hive-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/mixed-hive-format/</guid>
- <description>Mixed-Hive Format Mixed-Hive format is a format that has better compatibility with Hive than Mixed-Iceberg format. Mixed-Hive format uses a Hive table as the BaseStore and an Iceberg table as the ChangeStore. Mixed-Hive format supports:
schema, partition, and types consistent with Hive format Using the Hive connector to read and write Mixed-Hive format tables as Hive tables Upgrading a Hive table in-place to a Mixed-Hive format table without data rewriting or migra [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Mixed-Iceberg</title>
<link>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/mixed-iceberg-format/</guid>
- <description>Mixed-Iceberg Format Compared with Iceberg format, Mixed-Iceberg format provides more features:
Stronger primary key constraints that also apply to Spark OLAP performance that is production-ready for real-time data warehouses through the auto-bucket mechanism LogStore configuration that can reduce data pipeline latency from minutes to milliseconds/seconds Transaction conflict resolution mechanism that enables concurrent writes with the same primary key The design i [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Overview</title>
<link>https://amoro.apache.org/docs/latest/formats-overview/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/formats-overview/</guid>
- <description>Formats Overview Table format (aka. format) was first proposed by Iceberg, which can be described as follows:
It defines the relationship between tables and files, and any engine can query and retrieve data files according to the table format. New formats such as Iceberg/Delta/Hudi further define the relationship between tables and snapshots, and the relationship between snapshots and files. All write operations on the table will generate new snapshots, and all rea [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Paimon</title>
<link>https://amoro.apache.org/docs/latest/paimon-format/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/paimon-format/</guid>
- <description>Paimon Format Paimon format refers to Apache Paimon table. Paimon is a streaming data lake platform with high-speed data ingestion, changelog tracking and efficient real-time analytics.
By registering Paimon&rsquo;s catalog with Amoro, users can view information such as Schema, Options, Files, Snapshots, DDLs, Compaction information, and more for Paimon tables. Furthermore, they can operate on Paimon tables using Spark SQL in the Terminal. The current supported [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>REST API</title>
<link>https://amoro.apache.org/docs/latest/rest-api/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/rest-api/</guid>
- <description> REST API Accessing Swagger UI Access the Swagger UI at:
http://&lt;host&gt;:&lt;port&gt;/#/openapi-ui You can also view the OpenAPI specification YAML in the repository at amoro-ams/src/main/resources/openapi/openapi.yaml, or use the Swagger Editor online.
Building the OpenAPI SDK Build the OpenAPI SDK using Maven:
./mvnw clean package -pl amoro-openapi-sdk -am -Popenapi-sdk </description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Self-Optimizing</title>
<link>https://amoro.apache.org/docs/latest/self-optimizing/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/self-optimizing/</guid>
- <description>Self-optimizing Introduction Lakehouse is characterized by its openness and loose coupling, with data and files maintained by users through various engines. While this architecture appears to be well-suited for T+1 scenarios, as more attention is paid to applying Lakehouse to streaming data warehouses and real-time analysis scenarios, challenges arise. For example:
Streaming writes bring a massive amount of fragment files CDC ingestion and streaming updates generat [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Configuration</title>
<link>https://amoro.apache.org/docs/latest/spark-configuration/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-configuration/</guid>
- <description>Spark Configuration Catalogs configuration Using Mixed-Format in a standalone catalog Starting from version 3.x, Spark supports configuring an independent Catalog. If you want to use a Mixed-Format table in a standalone Catalog, you can create a mixed_catalog and load catalog metadata from AMS with following properties:
spark.sql.catalog.mixed_catalog=org.apache.amoro.spark.MixedFormatSparkCatalog spark.sql.catalog.mixed_catalog.ams.uri=thrift://${AMS_HOST}:${AMS_P [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark DDL</title>
<link>https://amoro.apache.org/docs/latest/spark-ddl/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-ddl/</guid>
- <description>Spark DDL CREATE TABLE To create an MixedFormat table under an Amoro Catalog, you can use using mixed_iceberg or using mixed_hive to specify the provider in the CREATE TABLE statement. If the Catalog type is Hive, the created table will be a Hive-compatible table.
CREATE TABLE mixed_catalog.db.sample ( id bigint COMMENT &#34;unique id&#34;, data string ) USING mixed_iceberg PRIMARY KEY You can use PRIMARY KEY in the CREATE TABLE statement to specify the pri [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Getting Started</title>
<link>https://amoro.apache.org/docs/latest/spark-getting-started/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-getting-started/</guid>
- <description>Spark Getting Started Iceberg Format The Iceberg Format can be accessed using the Connector provided by Iceberg. Refer to the documentation at Iceberg Spark Connector for more information.
Paimon Format The Paimon Format can be accessed using the Connector provided by Paimon. Refer to the documentation at Paimon Spark Connector for more information.
Mixed Format To use Amoro in a Spark shell, use the &ndash;packages option:
spark-shell --packages org.ap [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Queries</title>
<link>https://amoro.apache.org/docs/latest/spark-queries/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-queries/</guid>
- <description>Spark Queries Querying with SQL Querying Mixed-Format table by merge on read Using Select statement to query on Mixed-Format tables.
SELECT * FROM mixed_catalog.db.sample The Mixed-Format connector will merge the data from BaseStore and ChangeStore.
Query on change store For a Mixed-Format table with primary keys. you can query on ChangeStore by .change.
SELECT * FROM mixed_catalog.db.sample.change +---+----+----+---------------+------------+--------------+ [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Spark Writes</title>
<link>https://amoro.apache.org/docs/latest/spark-writes/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/spark-writes/</guid>
- <description>Spark Writes Writing with SQL INSERT OVERWRITE INSERT OVERWRITE can replace the partition in a table with the results of a query.
The default overwrite mode of Spark is Static, you can change the overwrite mode by
SET spark.sql.sources.partitionOverwriteMode=dynamic To demonstrate the behavior of dynamic and static overwrites, a test table is defined using the following DDL:
CREATE TABLE mixed_catalog.db.sample ( id int, data string, ts timestamp, primary k [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Table Watermark</title>
<link>https://amoro.apache.org/docs/latest/table-watermark/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/table-watermark/</guid>
- <description>Table Watermark Table freshness Data freshness represents timeliness, and in many discussions, freshness is considered one of the important indicators of data quality. In traditional offline data warehouses, higher cost typically means better performance, creating a typical binary paradox in terms of cost-performance trade-off. However, in high-freshness streaming data warehouses, massive small files and frequent updates can lead to performance degradation. The higher [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Trino</title>
<link>https://amoro.apache.org/docs/latest/trino/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/trino/</guid>
- <description>Trino Iceberg format Iceberg format can be accessed using the Iceberg Connector provided by Trino. please refer to the documentation at Iceberg Trino user manual for more information.
Paimon format Paimon format can be accessed using the Paimon Connector provided by Trino. please refer to the documentation at Paimon Trino user manual for more information.
Mixed format Install Create the {trino_home}/plugin/amoro directory in the Trino installation package, and [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Customized Encryption Method for Configurations</title>
<link>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-customized-encryption-method/</guid>
- <description>Using Customized Encryption Method for Configurations To enhance security, AMS allows encrypted sensitive configuration items such as passwords. Currently, AMS only supports the built-in base64 encryption algorithm (see Configure encrypted configuration items for details). If you require a stronger or customized encryption method, AMS also provides the flexibility to implement your own encryption algorithm.
Develop the Custom Implementation To integrate a custom en [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Kyuubi By Terminal</title>
<link>https://amoro.apache.org/docs/latest/using-kyuubi/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-kyuubi/</guid>
- <description>Using Kyuubi By Terminal Prerequisites:
There must be a running Kyuubi. To deploy and run Kyuubi, please refer to Kyuubi doc Terminal supports interfacing with Kyuubi to submit SQL to Kyuubi for execution. All you need to do is add the Kyuubi configuration as instructed below:
ams: terminal: backend: kyuubi kyuubi.jdbc.url: jdbc:hive2://127.0.0.1:10009/ # kyuubi Connection Address Kerberos Authentication Amoro terminal uses the Kerberos authentication informati [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Logstore</title>
<link>https://amoro.apache.org/docs/latest/flink-using-logstore/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/flink-using-logstore/</guid>
- <description>Using Logstore Due to the limitations of traditional offline data warehouse architectures in supporting real-time business needs, real-time data warehousing has experienced rapid evolution in recent years. In the architecture of real-time data warehousing, Apache Kafka is often used as the storage system for real-time data. However, this also brings about the issue of data disconnection between offline data warehouses.
Developers often need to pay attention to data [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Tables</title>
<link>https://amoro.apache.org/docs/latest/using-tables/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-tables/</guid>
- <description>Using Tables The SQL execution tool Terminal is provided in AMS dashboard to help users quickly create, modify and delete tables. It is also available in Spark and Flink and other engines to manage tables using SQL.
Create table After logging into AMS dashboard, go to Terminal, enter the table creation statement and execute it to complete the table creation. The following is an example of table creation:
create table test_db.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/managing-optimizers/index.html b/output/docs/latest/managing-optimizers/index.html
index 900d45551..e4c2a70bb 100644
--- a/output/docs/latest/managing-optimizers/index.html
+++ b/output/docs/latest/managing-optimizers/index.html
@@ -643,28 +643,28 @@
<p>Local container is a way to start Optimizer by local process and supports multi-threaded execution of Optimizer tasks. It is recommended to be used only in demo or local deployment scenarios. If the environment variable for jdk is not configured, the user can configure java_home to point to the jdk root directory. If already configured, this configuration item can be ignored.</p>
<p>Local container support the following properties:</p>
<table>
-<thead>
-<tr>
-<th>Property Name</th>
-<th>Required</th>
-<th>Default Value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>ams-optimizing-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>URI of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
-</tr>
-<tr>
-<td>export.JAVA_HOME</td>
-<td>false</td>
-<td>N/A</td>
-<td>Java runtime location</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property Name</th>
+ <th>Required</th>
+ <th>Default Value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>ams-optimizing-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>URI of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
+ </tr>
+ <tr>
+ <td>export.JAVA_HOME</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Java runtime location</td>
+ </tr>
+ </tbody>
</table>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">containers</span>:
</span></span><span style="display:flex;"><span> - <span style="color:#f92672">name</span>: <span style="color:#ae81ff">localContainer</span>
@@ -674,55 +674,55 @@
</span></span></code></pre></div><p>The format for optimizing URI is <code>thrift://{host}:{port}?parameter1=value2&parameter2=value2</code>.
The supported parameters include:</p>
<table>
-<thead>
-<tr>
-<th>Parameter Name</th>
-<th>Default Value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>autoReconnect</td>
-<td>true</td>
-<td>If reconnect the server when the connection is broken</td>
-</tr>
-<tr>
-<td>maxReconnects</td>
-<td>5</td>
-<td>Retry times when reconnecting</td>
-</tr>
-<tr>
-<td>connectTimeout</td>
-<td>0 (Forever)</td>
-<td>Timeout in milliseconds when connecting the server</td>
-</tr>
-<tr>
-<td>socketTimeout</td>
-<td>0 (Forever)</td>
-<td>Timeout in milliseconds when communicating with the server</td>
-</tr>
-<tr>
-<td>maxMessageSize</td>
-<td>104856600 (100MB)</td>
-<td>Max message size when communicating with the server</td>
-</tr>
-<tr>
-<td>maxMessageSize</td>
-<td>104856600 (100MB)</td>
-<td>Max message size when communicating with the server</td>
-</tr>
-<tr>
-<td>minIdle</td>
-<td>0</td>
-<td>Minimal idle clients in the pool</td>
-</tr>
-<tr>
-<td>maxIdle</td>
-<td>5</td>
-<td>Maximal idle clients in the pool</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Parameter Name</th>
+ <th>Default Value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>autoReconnect</td>
+ <td>true</td>
+ <td>If reconnect the server when the connection is broken</td>
+ </tr>
+ <tr>
+ <td>maxReconnects</td>
+ <td>5</td>
+ <td>Retry times when reconnecting</td>
+ </tr>
+ <tr>
+ <td>connectTimeout</td>
+ <td>0 (Forever)</td>
+ <td>Timeout in milliseconds when connecting the server</td>
+ </tr>
+ <tr>
+ <td>socketTimeout</td>
+ <td>0 (Forever)</td>
+ <td>Timeout in milliseconds when communicating with the server</td>
+ </tr>
+ <tr>
+ <td>maxMessageSize</td>
+ <td>104856600 (100MB)</td>
+ <td>Max message size when communicating with the server</td>
+ </tr>
+ <tr>
+ <td>maxMessageSize</td>
+ <td>104856600 (100MB)</td>
+ <td>Max message size when communicating with the server</td>
+ </tr>
+ <tr>
+ <td>minIdle</td>
+ <td>0</td>
+ <td>Minimal idle clients in the pool</td>
+ </tr>
+ <tr>
+ <td>maxIdle</td>
+ <td>5</td>
+ <td>Maximal idle clients in the pool</td>
+ </tr>
+ </tbody>
</table>
<h3 id="kubernetes-container">Kubernetes container</h3>
<p>Kubernetes container is a way to start Optimizer On K8s with standalone Optimizer.
@@ -730,64 +730,64 @@ To use Kubernetes container, you need to add a new container configuration.
with container-impl as <code>org.apache.amoro.server.manager.KubernetesOptimizerContainer</code></p>
<p>Kubernetes container support the following properties:</p>
<table>
-<thead>
-<tr>
-<th>Property Name</th>
-<th>Required</th>
-<th>Default Value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>kube-config-path</td>
-<td>true</td>
-<td>N/A</td>
-<td>Kubernetes config location</td>
-</tr>
-<tr>
-<td>image</td>
-<td>true</td>
-<td>N/A</td>
-<td>Optimizer Image name</td>
-</tr>
-<tr>
-<td>pullPolicy</td>
-<td>false</td>
-<td>IfNotPresent</td>
-<td>Specify the imagePullPolicy in the container spec</td>
-</tr>
-<tr>
-<td>namespace</td>
-<td>false</td>
-<td>“default”</td>
-<td>The namespace of optimizer to deploy</td>
-</tr>
-<tr>
-<td>ams-optimizing-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>URI of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
-</tr>
-<tr>
-<td>cpu.factor</td>
-<td>false</td>
-<td>“1.0”</td>
-<td>CPU factor when request kubernetes resource. Default 1 Cpu pre thread</td>
-</tr>
-<tr>
-<td>memory</td>
-<td>true</td>
-<td>N/A</td>
-<td>Memory usage for pre thread</td>
-</tr>
-<tr>
-<td>extra.jvm.heap.ratio</td>
-<td>false</td>
-<td>0.8</td>
-<td>The ratio of JVM heap memory to total pod memory</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property Name</th>
+ <th>Required</th>
+ <th>Default Value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>kube-config-path</td>
+ <td>true</td>
+ <td>N/A</td>
+ <td>Kubernetes config location</td>
+ </tr>
+ <tr>
+ <td>image</td>
+ <td>true</td>
+ <td>N/A</td>
+ <td>Optimizer Image name</td>
+ </tr>
+ <tr>
+ <td>pullPolicy</td>
+ <td>false</td>
+ <td>IfNotPresent</td>
+ <td>Specify the imagePullPolicy in the container spec</td>
+ </tr>
+ <tr>
+ <td>namespace</td>
+ <td>false</td>
+ <td>“default”</td>
+ <td>The namespace of optimizer to deploy</td>
+ </tr>
+ <tr>
+ <td>ams-optimizing-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>URI of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
+ </tr>
+ <tr>
+ <td>cpu.factor</td>
+ <td>false</td>
+ <td>“1.0”</td>
+ <td>CPU factor when request kubernetes resource. Default 1 Cpu pre thread</td>
+ </tr>
+ <tr>
+ <td>memory</td>
+ <td>true</td>
+ <td>N/A</td>
+ <td>Memory usage for pre thread</td>
+ </tr>
+ <tr>
+ <td>extra.jvm.heap.ratio</td>
+ <td>false</td>
+ <td>0.8</td>
+ <td>The ratio of JVM heap memory to total pod memory</td>
+ </tr>
+ </tbody>
</table>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">containers</span>:
</span></span><span style="display:flex;"><span> - <span style="color:#f92672">name</span>: <span style="color:#ae81ff">KubernetesContainer</span>
@@ -802,82 +802,82 @@ on yarn clusters or kubernetes clusters to support large-scale data scenarios. T
you need to add a new container configuration. with container-impl as <code>org.apache.amoro.server.manager.FlinkOptimizerContainer</code></p>
<p>Flink container support the following properties:</p>
<table>
-<thead>
-<tr>
-<th>Property Name</th>
-<th>Required</th>
-<th>Default Value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>flink-home</td>
-<td>true</td>
-<td>N/A</td>
-<td>Flink installation location</td>
-</tr>
-<tr>
-<td>target</td>
-<td>true</td>
-<td>yarn-per-job</td>
-<td>flink job deployed target, available values <code>yarn-per-job</code>, <code>yarn-application</code>, <code>kubernetes-application</code>, <code>session</code></td>
-</tr>
-<tr>
-<td>job-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>The jar uri of flink optimizer job. This is required if target is application mode.</td>
-</tr>
-<tr>
-<td>ams-optimizing-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>uri of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
-</tr>
-<tr>
-<td>export.<key></td>
-<td>false</td>
-<td>N/A</td>
-<td>environment variables will be exported during job submit</td>
-</tr>
-<tr>
-<td>export.JAVA_HOME</td>
-<td>false</td>
-<td>N/A</td>
-<td>Java runtime location</td>
-</tr>
-<tr>
-<td>export.HADOOP_CONF_DIR</td>
-<td>false</td>
-<td>N/A</td>
-<td>Direction which holds the configuration files for the hadoop cluster (including hdfs-site.xml, core-site.xml, yarn-site.xml ). If the hadoop cluster has kerberos authentication enabled, you need to prepare an additional krb5.conf and a keytab file for the user to submit tasks</td>
-</tr>
-<tr>
-<td>export.JVM_ARGS</td>
-<td>false</td>
-<td>N/A</td>
-<td>you can configure flink to run additional configuration parameters, here is an example of configuring krb5.conf, specify the address of krb5.conf to be used by Flink when committing via <code>-Djava.security.krb5.conf=/opt/krb5.conf</code></td>
-</tr>
-<tr>
-<td>export.HADOOP_USER_NAME</td>
-<td>false</td>
-<td>N/A</td>
-<td>the username used to submit tasks to yarn, used for simple authentication</td>
-</tr>
-<tr>
-<td>export.FLINK_CONF_DIR</td>
-<td>false</td>
-<td>N/A</td>
-<td>the directory where flink_conf.yaml is located</td>
-</tr>
-<tr>
-<td>flink-conf.<key></td>
-<td>false</td>
-<td>N/A</td>
-<td><a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/">Flink Configuration Options</a> will be passed to cli by <code>-Dkey=value</code>,</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property Name</th>
+ <th>Required</th>
+ <th>Default Value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>flink-home</td>
+ <td>true</td>
+ <td>N/A</td>
+ <td>Flink installation location</td>
+ </tr>
+ <tr>
+ <td>target</td>
+ <td>true</td>
+ <td>yarn-per-job</td>
+ <td>flink job deployed target, available values <code>yarn-per-job</code>, <code>yarn-application</code>, <code>kubernetes-application</code>, <code>session</code></td>
+ </tr>
+ <tr>
+ <td>job-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>The jar uri of flink optimizer job. This is required if target is application mode.</td>
+ </tr>
+ <tr>
+ <td>ams-optimizing-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>uri of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
+ </tr>
+ <tr>
+ <td>export.<key></td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>environment variables will be exported during job submit</td>
+ </tr>
+ <tr>
+ <td>export.JAVA_HOME</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Java runtime location</td>
+ </tr>
+ <tr>
+ <td>export.HADOOP_CONF_DIR</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Direction which holds the configuration files for the hadoop cluster (including hdfs-site.xml, core-site.xml, yarn-site.xml ). If the hadoop cluster has kerberos authentication enabled, you need to prepare an additional krb5.conf and a keytab file for the user to submit tasks</td>
+ </tr>
+ <tr>
+ <td>export.JVM_ARGS</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>you can configure flink to run additional configuration parameters, here is an example of configuring krb5.conf, specify the address of krb5.conf to be used by Flink when committing via <code>-Djava.security.krb5.conf=/opt/krb5.conf</code></td>
+ </tr>
+ <tr>
+ <td>export.HADOOP_USER_NAME</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>the username used to submit tasks to yarn, used for simple authentication</td>
+ </tr>
+ <tr>
+ <td>export.FLINK_CONF_DIR</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>the directory where flink_conf.yaml is located</td>
+ </tr>
+ <tr>
+ <td>flink-conf.<key></td>
+ <td>false</td>
+ <td>N/A</td>
+ <td><a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/">Flink Configuration Options</a> will be passed to cli by <code>-Dkey=value</code>,</td>
+ </tr>
+ </tbody>
</table>
<div class="info">
<p>To better utilize the resources of Flink Optimizer, it is recommended to add the following configuration to the Flink Optimizer Group:</p>
@@ -931,88 +931,88 @@ on yarn clusters or kubernetes clusters to support large-scale data scenarios. T
you need to add a new container configuration. with container-impl as <code>org.apache.amoro.server.manager.SparkOptimizerContainer</code></p>
<p>Spark container support the following properties:</p>
<table>
-<thead>
-<tr>
-<th>Property Name</th>
-<th>Required</th>
-<th>Default Value</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>spark-home</td>
-<td>true</td>
-<td>N/A</td>
-<td>Spark installation location</td>
-</tr>
-<tr>
-<td>master</td>
-<td>true</td>
-<td>yarn</td>
-<td>The cluster manager to connect to, available values <code>yarn</code>, <code>k8s://HOST:PORT</code></td>
-</tr>
-<tr>
-<td>deploy-mode</td>
-<td>true</td>
-<td>client</td>
-<td>Spark job deploy mode, available values <code>client</code>, <code>cluster</code></td>
-</tr>
-<tr>
-<td>job-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>The jar uri of spark optimizer job. This is required if deploy mode is cluster mode.</td>
-</tr>
-<tr>
-<td>ams-optimizing-uri</td>
-<td>false</td>
-<td>N/A</td>
-<td>uri of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
-</tr>
-<tr>
-<td>export.<key></td>
-<td>false</td>
-<td>N/A</td>
-<td>Environment variables will be exported during job submit</td>
-</tr>
-<tr>
-<td>export.JAVA_HOME</td>
-<td>false</td>
-<td>N/A</td>
-<td>Java runtime location</td>
-</tr>
-<tr>
-<td>export.HADOOP_CONF_DIR</td>
-<td>false</td>
-<td>N/A</td>
-<td>Direction which holds the configuration files for the hadoop cluster (including hdfs-site.xml, core-site.xml, yarn-site.xml ). If the hadoop cluster has kerberos authentication enabled, you need to prepare an additional krb5.conf and a keytab file for the user to submit tasks</td>
-</tr>
-<tr>
-<td>export.JVM_ARGS</td>
-<td>false</td>
-<td>N/A</td>
-<td>You can configure spark to run additional configuration parameters, here is an example of configuring krb5.conf, specify the address of krb5.conf to be used by Spark when committing via <code>-Djava.security.krb5.conf=/opt/krb5.conf</code></td>
-</tr>
-<tr>
-<td>export.HADOOP_USER_NAME</td>
-<td>false</td>
-<td>N/A</td>
-<td>The username used to submit tasks to yarn, used for simple authentication</td>
-</tr>
-<tr>
-<td>export.SPARK_CONF_DIR</td>
-<td>false</td>
-<td>N/A</td>
-<td>The directory where spark_conf.yaml is located</td>
-</tr>
-<tr>
-<td>spark-conf.<key></td>
-<td>false</td>
-<td>N/A</td>
-<td><a href="https://spark.apache.org/docs/latest/configuration.html">Spark Configuration Options</a> will be passed to cli by <code>-conf key=value</code>,</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property Name</th>
+ <th>Required</th>
+ <th>Default Value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>spark-home</td>
+ <td>true</td>
+ <td>N/A</td>
+ <td>Spark installation location</td>
+ </tr>
+ <tr>
+ <td>master</td>
+ <td>true</td>
+ <td>yarn</td>
+ <td>The cluster manager to connect to, available values <code>yarn</code>, <code>k8s://HOST:PORT</code></td>
+ </tr>
+ <tr>
+ <td>deploy-mode</td>
+ <td>true</td>
+ <td>client</td>
+ <td>Spark job deploy mode, available values <code>client</code>, <code>cluster</code></td>
+ </tr>
+ <tr>
+ <td>job-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>The jar uri of spark optimizer job. This is required if deploy mode is cluster mode.</td>
+ </tr>
+ <tr>
+ <td>ams-optimizing-uri</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>uri of AMS thrift self-optimizing endpoint. This could be used if the ams.server-expose-host is not available</td>
+ </tr>
+ <tr>
+ <td>export.<key></td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Environment variables will be exported during job submit</td>
+ </tr>
+ <tr>
+ <td>export.JAVA_HOME</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Java runtime location</td>
+ </tr>
+ <tr>
+ <td>export.HADOOP_CONF_DIR</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>Direction which holds the configuration files for the hadoop cluster (including hdfs-site.xml, core-site.xml, yarn-site.xml ). If the hadoop cluster has kerberos authentication enabled, you need to prepare an additional krb5.conf and a keytab file for the user to submit tasks</td>
+ </tr>
+ <tr>
+ <td>export.JVM_ARGS</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>You can configure spark to run additional configuration parameters, here is an example of configuring krb5.conf, specify the address of krb5.conf to be used by Spark when committing via <code>-Djava.security.krb5.conf=/opt/krb5.conf</code></td>
+ </tr>
+ <tr>
+ <td>export.HADOOP_USER_NAME</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>The username used to submit tasks to yarn, used for simple authentication</td>
+ </tr>
+ <tr>
+ <td>export.SPARK_CONF_DIR</td>
+ <td>false</td>
+ <td>N/A</td>
+ <td>The directory where spark_conf.yaml is located</td>
+ </tr>
+ <tr>
+ <td>spark-conf.<key></td>
+ <td>false</td>
+ <td>N/A</td>
+ <td><a href="https://spark.apache.org/docs/latest/configuration.html">Spark Configuration Options</a> will be passed to cli by <code>-conf key=value</code>,</td>
+ </tr>
+ </tbody>
</table>
<div class="info">
<p>To better utilize the resources of Spark Optimizer, the DRA(Dynamic Resource Allocation) feature is switched on, and the optimizer parallelism equals `spark.dynamicAllocation.maxExecutors.
@@ -1080,87 +1080,87 @@ contain several optimizers with the same container implementation to facilitate
</ul>
<p>The optimizer group supports the following properties:</p>
<table>
-<thead>
-<tr>
-<th>Property</th>
-<th>Container type</th>
-<th>Required</th>
-<th>Default</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>scheduling-policy</td>
-<td>All</td>
-<td>No</td>
-<td>quota</td>
-<td>The scheduler group scheduling policy, the default value is <code>quota</code>, it will be scheduled according to the quota resources configured for each table, the larger the table quota is, the more optimizer resources it can take. There is also a configuration <code>balanced</code> that will balance the scheduling of each table, the longer the table has not been optimized, the higher the scheduling priority will be.</td>
-</tr>
-<tr>
-<td>max-input-file-size-per-thread</td>
-<td>All</td>
-<td>No</td>
-<td>536870912(512MB)</td>
-<td>Max input file size per optimize thread.</td>
-</tr>
-<tr>
-<td>ams-optimizing-uri</td>
-<td>All</td>
-<td>No</td>
-<td>thrift://{ams.server-expose-host}:{ams.thrift-server.optimizing-service.binding-port}</td>
-<td>Table optimizing service endpoint. This is used when the default service endpoint is not visitable.</td>
-</tr>
-<tr>
-<td>cache-enabled</td>
-<td>All</td>
-<td>No</td>
-<td>false</td>
-<td>Whether enable cache in optimizer.</td>
-</tr>
-<tr>
-<td>cache-max-total-size</td>
-<td>All</td>
-<td>No</td>
-<td>128mb</td>
-<td>Max total size in optimier cache.</td>
-</tr>
-<tr>
-<td>cache-max-entry-size</td>
-<td>All</td>
-<td>No</td>
-<td>64mb</td>
-<td>Max entry size in optimizer cache.</td>
-</tr>
-<tr>
-<td>cache-timeout</td>
-<td>All</td>
-<td>No</td>
-<td>10min</td>
-<td>Timeout in optimizer cache.</td>
-</tr>
-<tr>
-<td>memory</td>
-<td>Local</td>
-<td>Yes</td>
-<td>N/A</td>
-<td>The max memory of JVM for local optimizer, in MBs.</td>
-</tr>
-<tr>
-<td>flink-conf.<key></td>
-<td>Flink</td>
-<td>No</td>
-<td>N/A</td>
-<td>Any flink config options could be overwritten, priority is optimizing-group > optimizing-container > flink-conf.yaml.</td>
-</tr>
-<tr>
-<td>spark-conf.<key></td>
-<td>Spark</td>
-<td>No</td>
-<td>N/A</td>
-<td>Any spark config options could be overwritten, priority is optimizing-group > optimizing-container > spark-defaults.conf.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property</th>
+ <th>Container type</th>
+ <th>Required</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>scheduling-policy</td>
+ <td>All</td>
+ <td>No</td>
+ <td>quota</td>
+ <td>The scheduler group scheduling policy, the default value is <code>quota</code>, it will be scheduled according to the quota resources configured for each table, the larger the table quota is, the more optimizer resources it can take. There is also a configuration <code>balanced</code> that will balance the scheduling of each table, the longer the table has not been optimized, the higher the scheduling priority will be.</td>
+ </tr>
+ <tr>
+ <td>max-input-file-size-per-thread</td>
+ <td>All</td>
+ <td>No</td>
+ <td>536870912(512MB)</td>
+ <td>Max input file size per optimize thread.</td>
+ </tr>
+ <tr>
+ <td>ams-optimizing-uri</td>
+ <td>All</td>
+ <td>No</td>
+ <td>thrift://{ams.server-expose-host}:{ams.thrift-server.optimizing-service.binding-port}</td>
+ <td>Table optimizing service endpoint. This is used when the default service endpoint is not visitable.</td>
+ </tr>
+ <tr>
+ <td>cache-enabled</td>
+ <td>All</td>
+ <td>No</td>
+ <td>false</td>
+ <td>Whether enable cache in optimizer.</td>
+ </tr>
+ <tr>
+ <td>cache-max-total-size</td>
+ <td>All</td>
+ <td>No</td>
+ <td>128mb</td>
+ <td>Max total size in optimier cache.</td>
+ </tr>
+ <tr>
+ <td>cache-max-entry-size</td>
+ <td>All</td>
+ <td>No</td>
+ <td>64mb</td>
+ <td>Max entry size in optimizer cache.</td>
+ </tr>
+ <tr>
+ <td>cache-timeout</td>
+ <td>All</td>
+ <td>No</td>
+ <td>10min</td>
+ <td>Timeout in optimizer cache.</td>
+ </tr>
+ <tr>
+ <td>memory</td>
+ <td>Local</td>
+ <td>Yes</td>
+ <td>N/A</td>
+ <td>The max memory of JVM for local optimizer, in MBs.</td>
+ </tr>
+ <tr>
+ <td>flink-conf.<key></td>
+ <td>Flink</td>
+ <td>No</td>
+ <td>N/A</td>
+ <td>Any flink config options could be overwritten, priority is optimizing-group > optimizing-container > flink-conf.yaml.</td>
+ </tr>
+ <tr>
+ <td>spark-conf.<key></td>
+ <td>Spark</td>
+ <td>No</td>
+ <td>N/A</td>
+ <td>Any spark config options could be overwritten, priority is optimizing-group > optimizing-container > spark-defaults.conf.</td>
+ </tr>
+ </tbody>
</table>
<div class="info">
<p>To better utilize the resources of Flink Optimizer, it is recommended to add the following configuration to the Flink Optimizer Group:</p>
@@ -1190,160 +1190,160 @@ If the optimizer runs normally, you will see an optimizer with the status <code>
<h3 id="deploy-external-optimizer">Deploy external optimizer</h3>
<p>You can submit optimizer in your own Flink task development platform or local Flink environment with the following configuration. The main parameters include:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>./bin/flink run-application -t yarn-application <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -Djobmanager.memory.process.size<span style="color:#f92672">=</span>1024mb <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -Dtaskmanager.memory.process.size<span style="color:#f92672">=</span>2048mb <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -Dtaskmanager.memory.managed.size<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -Dtaskmanager.memory.network.max<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -Dtaskmanager.memory.network.min<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -c org.apache.amoro.optimizer.flink.FlinkOptimizer <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> <span style="color:#e6db74">${</span>AMORO_HOME<span style="color:#e6db74">}</span>/plugin/optimizer/flink/optimizer-job.jar <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -a thrift://127.0.0.1:1261 <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -g flinkGroup <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -p <span style="color:#ae81ff">1</span>
+</span></span></span><span style="display:flex;"><span> -Djobmanager.memory.process.size<span style="color:#f92672">=</span>1024mb <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -Dtaskmanager.memory.process.size<span style="color:#f92672">=</span>2048mb <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -Dtaskmanager.memory.managed.size<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -Dtaskmanager.memory.network.max<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -Dtaskmanager.memory.network.min<span style="color:#f92672">=</span>32mb <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -c org.apache.amoro.optimizer.flink.FlinkOptimizer <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> <span style="color:#e6db74">${</span>AMORO_HOME<span style="color:#e6db74">}</span>/plugin/optimizer/flink/optimizer-job.jar <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -a thrift://127.0.0.1:1261 <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -g flinkGroup <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -p <span style="color:#ae81ff">1</span>
</span></span></code></pre></div><p>The description of the relevant parameters is shown in the following table:</p>
<table>
-<thead>
-<tr>
-<th>Property</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>-a</td>
-<td>Yes</td>
-<td>The address of the AMS thrift service, for example: thrift://127.0.0.1:1261, can be obtained from the config.yaml configuration.</td>
-</tr>
-<tr>
-<td>-g</td>
-<td>Yes</td>
-<td>Group name created in advance under external container.</td>
-</tr>
-<tr>
-<td>-p</td>
-<td>Yes</td>
-<td>Optimizer parallelism usage.</td>
-</tr>
-<tr>
-<td>-hb</td>
-<td>No</td>
-<td>Heart beat interval with ams, should be smaller than configuration ams.optimizer.heart-beat-timeout in AMS configuration conf/config.yaml which is 60000 milliseconds by default, default 10000(ms).</td>
-</tr>
-<tr>
-<td>-eds</td>
-<td>No</td>
-<td>Whether extend storage to disk, default false.</td>
-</tr>
-<tr>
-<td>-dsp</td>
-<td>No</td>
-<td>Defines the directory where the storage files are saved, the default temporary-file directory is specified by the system property <code>java.io.tmpdir</code>. On UNIX systems the default value of this property is typically “/tmp” or “/var/tmp”.</td>
-</tr>
-<tr>
-<td>-msz</td>
-<td>No</td>
-<td>Memory storage size limit when extending disk storage(MB), default 512(MB).</td>
-</tr>
-<tr>
-<td>-ce</td>
-<td>No</td>
-<td>Whether enable cache in optimizer, default false.</td>
-</tr>
-<tr>
-<td>-cmts</td>
-<td>No</td>
-<td>Max total size in optimier cache, default 128MB.</td>
-</tr>
-<tr>
-<td>-cmes</td>
-<td>No</td>
-<td>Max entry size in optimizer cache, default 64MB.</td>
-</tr>
-<tr>
-<td>-ct</td>
-<td>No</td>
-<td>Timeout in optimizer cache, default 10Min.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>-a</td>
+ <td>Yes</td>
+ <td>The address of the AMS thrift service, for example: thrift://127.0.0.1:1261, can be obtained from the config.yaml configuration.</td>
+ </tr>
+ <tr>
+ <td>-g</td>
+ <td>Yes</td>
+ <td>Group name created in advance under external container.</td>
+ </tr>
+ <tr>
+ <td>-p</td>
+ <td>Yes</td>
+ <td>Optimizer parallelism usage.</td>
+ </tr>
+ <tr>
+ <td>-hb</td>
+ <td>No</td>
+ <td>Heart beat interval with ams, should be smaller than configuration ams.optimizer.heart-beat-timeout in AMS configuration conf/config.yaml which is 60000 milliseconds by default, default 10000(ms).</td>
+ </tr>
+ <tr>
+ <td>-eds</td>
+ <td>No</td>
+ <td>Whether extend storage to disk, default false.</td>
+ </tr>
+ <tr>
+ <td>-dsp</td>
+ <td>No</td>
+ <td>Defines the directory where the storage files are saved, the default temporary-file directory is specified by the system property <code>java.io.tmpdir</code>. On UNIX systems the default value of this property is typically “/tmp” or “/var/tmp”.</td>
+ </tr>
+ <tr>
+ <td>-msz</td>
+ <td>No</td>
+ <td>Memory storage size limit when extending disk storage(MB), default 512(MB).</td>
+ </tr>
+ <tr>
+ <td>-ce</td>
+ <td>No</td>
+ <td>Whether enable cache in optimizer, default false.</td>
+ </tr>
+ <tr>
+ <td>-cmts</td>
+ <td>No</td>
+ <td>Max total size in optimier cache, default 128MB.</td>
+ </tr>
+ <tr>
+ <td>-cmes</td>
+ <td>No</td>
+ <td>Max entry size in optimizer cache, default 64MB.</td>
+ </tr>
+ <tr>
+ <td>-ct</td>
+ <td>No</td>
+ <td>Timeout in optimizer cache, default 10Min.</td>
+ </tr>
+ </tbody>
</table>
<p>Or you can submit optimizer in your own Spark task development platform or local Spark environment with the following configuration. The main parameters include:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>./bin/spark-submit --master yarn --deploy-mode cluster <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf <span style="color:#e6db74">"spark.driver.cores=1"</span> <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf <span style="color:#e6db74">"spark.driver.memory=g"</span> <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf <span style="color:#e6db74">"spark.executor.cores=1"</span> <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf <span style="color:#e6db74">"spark.executor.memory=2g"</span> <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --class org.apache.amoro.optimizer.spark.SparkOptimizer <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> <span style="color:#e6db74">${</span>AMORO_HOME<span style="color:#e6db74">}</span>/plugin/optimizer/spark/optimizer-job.jar <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -a thrift://127.0.0.1:1261 <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -g sparkGroup <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> -p <span style="color:#ae81ff">1</span>
+</span></span></span><span style="display:flex;"><span> --conf <span style="color:#e6db74">"spark.driver.cores=1"</span> <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf <span style="color:#e6db74">"spark.driver.memory=g"</span> <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf <span style="color:#e6db74">"spark.executor.cores=1"</span> <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf <span style="color:#e6db74">"spark.executor.memory=2g"</span> <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --class org.apache.amoro.optimizer.spark.SparkOptimizer <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> <span style="color:#e6db74">${</span>AMORO_HOME<span style="color:#e6db74">}</span>/plugin/optimizer/spark/optimizer-job.jar <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -a thrift://127.0.0.1:1261 <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -g sparkGroup <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> -p <span style="color:#ae81ff">1</span>
</span></span></code></pre></div><p>The description of the relevant parameters is shown in the following table:</p>
<table>
-<thead>
-<tr>
-<th>Property</th>
-<th>Required</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>-a</td>
-<td>Yes</td>
-<td>The address of the AMS thrift service, for example: thrift://127.0.0.1:1261, can be obtained from the config.yaml configuration.</td>
-</tr>
-<tr>
-<td>-g</td>
-<td>Yes</td>
-<td>Group name created in advance under external container.</td>
-</tr>
-<tr>
-<td>-p</td>
-<td>Yes</td>
-<td>Optimizer parallelism usage.</td>
-</tr>
-<tr>
-<td>-hb</td>
-<td>No</td>
-<td>Heart beat interval with ams, should be smaller than configuration ams.optimizer.heart-beat-timeout in AMS configuration conf/config.yaml which is 60000 milliseconds by default, default 10000(ms).</td>
-</tr>
-<tr>
-<td>-eds</td>
-<td>No</td>
-<td>Whether extend storage to disk, default false.</td>
-</tr>
-<tr>
-<td>-dsp</td>
-<td>No</td>
-<td>Defines the directory where the storage files are saved, the default temporary-file directory is specified by the system property <code>java.io.tmpdir</code>. On UNIX systems the default value of this property is typically “/tmp” or “/var/tmp”.</td>
-</tr>
-<tr>
-<td>-msz</td>
-<td>No</td>
-<td>Memory storage size limit when extending disk storage(MB), default 512(MB).</td>
-</tr>
-<tr>
-<td>-ce</td>
-<td>No</td>
-<td>Whether enable cache in optimizer, default false.</td>
-</tr>
-<tr>
-<td>-cmts</td>
-<td>No</td>
-<td>Max total size in optimier cache, default 128MB.</td>
-</tr>
-<tr>
-<td>-cmes</td>
-<td>No</td>
-<td>Max entry size in optimizer cache, default 64MB.</td>
-</tr>
-<tr>
-<td>-ct</td>
-<td>No</td>
-<td>Timeout in optimizer cache, default 10Min.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Property</th>
+ <th>Required</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>-a</td>
+ <td>Yes</td>
+ <td>The address of the AMS thrift service, for example: thrift://127.0.0.1:1261, can be obtained from the config.yaml configuration.</td>
+ </tr>
+ <tr>
+ <td>-g</td>
+ <td>Yes</td>
+ <td>Group name created in advance under external container.</td>
+ </tr>
+ <tr>
+ <td>-p</td>
+ <td>Yes</td>
+ <td>Optimizer parallelism usage.</td>
+ </tr>
+ <tr>
+ <td>-hb</td>
+ <td>No</td>
+ <td>Heart beat interval with ams, should be smaller than configuration ams.optimizer.heart-beat-timeout in AMS configuration conf/config.yaml which is 60000 milliseconds by default, default 10000(ms).</td>
+ </tr>
+ <tr>
+ <td>-eds</td>
+ <td>No</td>
+ <td>Whether extend storage to disk, default false.</td>
+ </tr>
+ <tr>
+ <td>-dsp</td>
+ <td>No</td>
+ <td>Defines the directory where the storage files are saved, the default temporary-file directory is specified by the system property <code>java.io.tmpdir</code>. On UNIX systems the default value of this property is typically “/tmp” or “/var/tmp”.</td>
+ </tr>
+ <tr>
+ <td>-msz</td>
+ <td>No</td>
+ <td>Memory storage size limit when extending disk storage(MB), default 512(MB).</td>
+ </tr>
+ <tr>
+ <td>-ce</td>
+ <td>No</td>
+ <td>Whether enable cache in optimizer, default false.</td>
+ </tr>
+ <tr>
+ <td>-cmts</td>
+ <td>No</td>
+ <td>Max total size in optimier cache, default 128MB.</td>
+ </tr>
+ <tr>
+ <td>-cmes</td>
+ <td>No</td>
+ <td>Max entry size in optimizer cache, default 64MB.</td>
+ </tr>
+ <tr>
+ <td>-ct</td>
+ <td>No</td>
+ <td>Timeout in optimizer cache, default 10Min.</td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/metrics/index.html b/output/docs/latest/metrics/index.html
index e470276c1..bd1dcce85 100644
--- a/output/docs/latest/metrics/index.html
+++ b/output/docs/latest/metrics/index.html
@@ -640,428 +640,428 @@
<p>Amoro has supported built-in metrics to measure status of table self-optimizing processes and optimizer resources, which can be <a href="../deployment/#configure-metric-reporter">reported to external metric system like Prometheus etc</a>.</p>
<h2 id="self-optimizing-metrics">Self-optimizing metrics</h2>
<table>
-<thead>
-<tr>
-<th>Metric Name</th>
-<th>Type</th>
-<th>Tags</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>table_optimizing_status_idle_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds after table be in idle status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_pending_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds after table be in pending status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_planning_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds after table be in planning status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_executing_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds after table be in executing status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_committing_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds after table be in committing status</td>
-</tr>
-<tr>
-<td>table_optimizing_process_total_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of all optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_process_failed_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of failed optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_minor_total_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of minor optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_minor_failed_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of failed minor optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_major_total_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of major optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_major_failed_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of failed major optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_full_total_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of full optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_full_failed_count</td>
-<td>Counter</td>
-<td>catalog, database, table, group</td>
-<td>Count of failed full optimizing process since ams started</td>
-</tr>
-<tr>
-<td>table_optimizing_status_in_idle</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>If currently table is in idle status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_in_pending</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>If currently table is in pending status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_in_planning</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>If currently table is in planning status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_in_executing</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>If currently table is in executing status</td>
-</tr>
-<tr>
-<td>table_optimizing_status_in_committing</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>If currently table is in committing status</td>
-</tr>
-<tr>
-<td>table_optimizing_since_last_minor_optimization_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds since last successful minor optimization</td>
-</tr>
-<tr>
-<td>table_optimizing_since_last_major_optimization_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds since last successful major optimization</td>
-</tr>
-<tr>
-<td>table_optimizing_since_last_full_optimization_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds since last successful full optimization</td>
-</tr>
-<tr>
-<td>table_optimizing_since_last_optimization_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds since last successful optimization</td>
-</tr>
-<tr>
-<td>table_optimizing_lag_duration_mills</td>
-<td>Gauge</td>
-<td>catalog, database, table, group</td>
-<td>Duration in milliseconds between last self-optimizing snapshot and refreshed snapshot</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Metric Name</th>
+ <th>Type</th>
+ <th>Tags</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>table_optimizing_status_idle_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds after table be in idle status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_pending_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds after table be in pending status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_planning_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds after table be in planning status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_executing_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds after table be in executing status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_committing_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds after table be in committing status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_process_total_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of all optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_process_failed_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of failed optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_minor_total_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of minor optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_minor_failed_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of failed minor optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_major_total_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of major optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_major_failed_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of failed major optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_full_total_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of full optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_full_failed_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table, group</td>
+ <td>Count of failed full optimizing process since ams started</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_in_idle</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>If currently table is in idle status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_in_pending</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>If currently table is in pending status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_in_planning</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>If currently table is in planning status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_in_executing</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>If currently table is in executing status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_status_in_committing</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>If currently table is in committing status</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_since_last_minor_optimization_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds since last successful minor optimization</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_since_last_major_optimization_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds since last successful major optimization</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_since_last_full_optimization_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds since last successful full optimization</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_since_last_optimization_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds since last successful optimization</td>
+ </tr>
+ <tr>
+ <td>table_optimizing_lag_duration_mills</td>
+ <td>Gauge</td>
+ <td>catalog, database, table, group</td>
+ <td>Duration in milliseconds between last self-optimizing snapshot and refreshed snapshot</td>
+ </tr>
+ </tbody>
</table>
<h2 id="optimizer-group-metrics">Optimizer Group metrics</h2>
<table>
-<thead>
-<tr>
-<th>Metric Name</th>
-<th>Type</th>
-<th>Tags</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>optimizer_group_pending_tasks</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of pending tasks in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_executing_tasks</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of executing tasks in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_planing_tables</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of planing tables in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_pending_tables</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of pending tables in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_executing_tables</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of executing tables in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_idle_tables</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of idle tables in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_committing_tables</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of committing tables in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_optimizer_instances</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of optimizer instances in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_memory_bytes_allocated</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Memory bytes allocated in optimizer group</td>
-</tr>
-<tr>
-<td>optimizer_group_threads</td>
-<td>Gauge</td>
-<td>group</td>
-<td>Number of total threads in optimizer group</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Metric Name</th>
+ <th>Type</th>
+ <th>Tags</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>optimizer_group_pending_tasks</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of pending tasks in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_executing_tasks</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of executing tasks in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_planing_tables</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of planing tables in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_pending_tables</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of pending tables in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_executing_tables</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of executing tables in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_idle_tables</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of idle tables in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_committing_tables</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of committing tables in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_optimizer_instances</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of optimizer instances in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_memory_bytes_allocated</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Memory bytes allocated in optimizer group</td>
+ </tr>
+ <tr>
+ <td>optimizer_group_threads</td>
+ <td>Gauge</td>
+ <td>group</td>
+ <td>Number of total threads in optimizer group</td>
+ </tr>
+ </tbody>
</table>
<h2 id="orphan-files-cleaning-metrics">Orphan Files Cleaning metrics</h2>
<table>
-<thead>
-<tr>
-<th>Metric Name</th>
-<th>Type</th>
-<th>Tags</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>table_orphan_content_file_cleaning_count</td>
-<td>Counter</td>
-<td>catalog, database, table</td>
-<td>Count of orphan content files cleaned in the table since ams started</td>
-</tr>
-<tr>
-<td>table_orphan_metadata_file_cleaning_count</td>
-<td>Counter</td>
-<td>catalog, database, table</td>
-<td>Count of orphan metadata files cleaned in the table since ams started</td>
-</tr>
-<tr>
-<td>table_expected_orphan_content_file_cleaning_count</td>
-<td>Counter</td>
-<td>catalog, database, table</td>
-<td>Expected Count of orphan content files cleaned in the table since ams started</td>
-</tr>
-<tr>
-<td>table_expected_orphan_metadata_file_cleaning_count</td>
-<td>Counter</td>
-<td>catalog, database, table</td>
-<td>Expected Count of orphan metadata files cleaned in the table since ams started</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Metric Name</th>
+ <th>Type</th>
+ <th>Tags</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>table_orphan_content_file_cleaning_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table</td>
+ <td>Count of orphan content files cleaned in the table since ams started</td>
+ </tr>
+ <tr>
+ <td>table_orphan_metadata_file_cleaning_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table</td>
+ <td>Count of orphan metadata files cleaned in the table since ams started</td>
+ </tr>
+ <tr>
+ <td>table_expected_orphan_content_file_cleaning_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table</td>
+ <td>Expected Count of orphan content files cleaned in the table since ams started</td>
+ </tr>
+ <tr>
+ <td>table_expected_orphan_metadata_file_cleaning_count</td>
+ <td>Counter</td>
+ <td>catalog, database, table</td>
+ <td>Expected Count of orphan metadata files cleaned in the table since ams started</td>
+ </tr>
+ </tbody>
</table>
<h2 id="ams-service-metrics">Ams service metrics</h2>
<table>
-<thead>
-<tr>
-<th>Metric Name</th>
-<th>Type</th>
-<th>Tags</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>ams_jvm_cpu_load</td>
-<td>Gauge</td>
-<td></td>
-<td>The recent CPU usage of the AMS</td>
-</tr>
-<tr>
-<td>ams_jvm_cpu_time</td>
-<td>Gauge</td>
-<td></td>
-<td>The CPU time used by the AMS</td>
-</tr>
-<tr>
-<td>ams_jvm_memory_heap_used</td>
-<td>Gauge</td>
-<td></td>
-<td>The amount of heap memory currently used (in bytes) by the AMS</td>
-</tr>
-<tr>
-<td>ams_jvm_memory_heap_committed</td>
-<td>Gauge</td>
-<td></td>
-<td>The amount of memory in the heap committed for JVM use (bytes)</td>
-</tr>
-<tr>
-<td>ams_jvm_memory_heap_max</td>
-<td>Gauge</td>
-<td></td>
-<td>The maximum heap memory (bytes), set by -Xmx JVM argument</td>
-</tr>
-<tr>
-<td>ams_jvm_threads_count</td>
-<td>Gauge</td>
-<td></td>
-<td>The total number of live threads used by the AMS</td>
-</tr>
-<tr>
-<td>ams_jvm_garbage_collector_count</td>
-<td>Gauge</td>
-<td>garbage_collector</td>
-<td>The count of the JVM’s Garbage Collector, such as G1 Young</td>
-</tr>
-<tr>
-<td>ams_jvm_garbage_collector_time</td>
-<td>Gauge</td>
-<td>garbage_collector</td>
-<td>The time spent by the JVM’s Garbage Collector, such as G1 Young</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Metric Name</th>
+ <th>Type</th>
+ <th>Tags</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>ams_jvm_cpu_load</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The recent CPU usage of the AMS</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_cpu_time</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The CPU time used by the AMS</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_memory_heap_used</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The amount of heap memory currently used (in bytes) by the AMS</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_memory_heap_committed</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The amount of memory in the heap committed for JVM use (bytes)</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_memory_heap_max</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The maximum heap memory (bytes), set by -Xmx JVM argument</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_threads_count</td>
+ <td>Gauge</td>
+ <td></td>
+ <td>The total number of live threads used by the AMS</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_garbage_collector_count</td>
+ <td>Gauge</td>
+ <td>garbage_collector</td>
+ <td>The count of the JVM’s Garbage Collector, such as G1 Young</td>
+ </tr>
+ <tr>
+ <td>ams_jvm_garbage_collector_time</td>
+ <td>Gauge</td>
+ <td>garbage_collector</td>
+ <td>The time spent by the JVM’s Garbage Collector, such as G1 Young</td>
+ </tr>
+ </tbody>
</table>
<h2 id="table-summary-metrics">table summary metrics</h2>
<table>
-<thead>
-<tr>
-<th>Metric Name</th>
-<th>Type</th>
-<th>Tags</th>
-<th>Description</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>table_summary_total_files</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Total number of files in the table</td>
-</tr>
-<tr>
-<td>table_summary_data_files</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Number of data files in the table</td>
-</tr>
-<tr>
-<td>table_summary_equality_delete_files</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Number of equality delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_position_delete_files</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Number of position delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_dangling_delete_files</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Number of dangling delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_total_files_size</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Total size of files in the table</td>
-</tr>
-<tr>
-<td>table_summary_data_files_size</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Size of data files in the table</td>
-</tr>
-<tr>
-<td>table_summary_equality_delete_files_size</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Size of equality delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_position_delete_files_size</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Size of position delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_total_records</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Total records in the table</td>
-</tr>
-<tr>
-<td>table_summary_data_files_records</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Records of data files in the table</td>
-</tr>
-<tr>
-<td>table_summary_equality_delete_files_records</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Records of equality delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_position_delete_files_records</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Records of position delete files in the table</td>
-</tr>
-<tr>
-<td>table_summary_snapshots</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Number of snapshots in the table</td>
-</tr>
-<tr>
-<td>table_summary_health_score</td>
-<td>Gauge</td>
-<td>catalog, database, table</td>
-<td>Health score of the table</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th>Metric Name</th>
+ <th>Type</th>
+ <th>Tags</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>table_summary_total_files</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Total number of files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_data_files</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Number of data files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_equality_delete_files</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Number of equality delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_position_delete_files</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Number of position delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_dangling_delete_files</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Number of dangling delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_total_files_size</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Total size of files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_data_files_size</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Size of data files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_equality_delete_files_size</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Size of equality delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_position_delete_files_size</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Size of position delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_total_records</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Total records in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_data_files_records</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Records of data files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_equality_delete_files_records</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Records of equality delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_position_delete_files_records</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Records of position delete files in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_snapshots</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Number of snapshots in the table</td>
+ </tr>
+ <tr>
+ <td>table_summary_health_score</td>
+ <td>Gauge</td>
+ <td>catalog, database, table</td>
+ <td>Health score of the table</td>
+ </tr>
+ </tbody>
</table>
</div>
diff --git a/output/docs/latest/mixed-iceberg-format/index.html b/output/docs/latest/mixed-iceberg-format/index.html
index 9ca2f0e95..0a54cbfdb 100644
--- a/output/docs/latest/mixed-iceberg-format/index.html
+++ b/output/docs/latest/mixed-iceberg-format/index.html
@@ -661,10 +661,10 @@ Auto-bucket divides the data under a partition into sets of non-intersecting pri
While possessing the features of the Mixed-Iceberg format, the BaseStore and ChangeStore can be read and written using the native Iceberg connector, thus having all the functional features of the Iceberg format.
Taking Spark as an example, this paper describes how to operate on the Mixed-Iceberg format table created by Quick demo using the Iceberg connector. We can use the following command to open a Spark SQL client:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>spark-sql --packages org.apache.Iceberg:Iceberg-spark-runtime-3.2_2.12:0.14.0<span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.sql.extensions<span style="color:#f92672">=</span>org.apache.Iceberg.spark.extensions.IcebergSparkSessionExtensions <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.sql.catalog.local<span style="color:#f92672">=</span>org.apache.Iceberg.spark.SparkCatalog <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.sql.catalog.local.type<span style="color:#f92672">=</span>hadoop <span style="color:#ae81ff">\
-</span></span></span><span style="display:flex;"><span><span style="color:#ae81ff"></span> --conf spark.sql.catalog.local.warehouse<span style="color:#f92672">=</span>/tmp/Amoro/warehouse
+</span></span></span><span style="display:flex;"><span> --conf spark.sql.extensions<span style="color:#f92672">=</span>org.apache.Iceberg.spark.extensions.IcebergSparkSessionExtensions <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf spark.sql.catalog.local<span style="color:#f92672">=</span>org.apache.Iceberg.spark.SparkCatalog <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf spark.sql.catalog.local.type<span style="color:#f92672">=</span>hadoop <span style="color:#ae81ff">\
+</span></span></span><span style="display:flex;"><span> --conf spark.sql.catalog.local.warehouse<span style="color:#f92672">=</span>/tmp/Amoro/warehouse
</span></span></code></pre></div><p>After that, we can use the following command to read from or write to the Iceberg tables managed by Amoro:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>-- Switch to Iceberg catalog
</span></span><span style="display:flex;"><span>use local;
diff --git a/output/docs/latest/search.json b/output/docs/latest/search.json
index 23bd186f1..6e3496e9d 100644
--- a/output/docs/latest/search.json
+++ b/output/docs/latest/search.json
@@ -1 +1 @@
-[{"categories":null,"content":"Amoro Docs This directory contains the documentation content of Amoro.\nThe documentation is written in Markdown format. The images referenced in the documentation are saved in the images folder. Style Proper nouns should start with a capital letter, like Hadoop、Hive、Iceberg、Amoro ","description":"","title":"","uri":"https://amoro.apache.org/docs/latest/readme/"},{"categories":null,"content":" AMS Configuration Amoro Management Service Configuration The con [...]
\ No newline at end of file
+[{"categories":null,"content":"Amoro Docs This directory contains the documentation content of Amoro.\nThe documentation is written in Markdown format. The images referenced in the documentation are saved in the images folder. Style Proper nouns should start with a capital letter, like Hadoop、Hive、Iceberg、Amoro ","description":"","title":"","uri":"https://amoro.apache.org/docs/latest/readme/"},{"categories":null,"content":" AMS Configuration Amoro Management Service Configuration The con [...]
\ No newline at end of file
diff --git a/output/docs/latest/self-optimizing/index.html b/output/docs/latest/self-optimizing/index.html
index c5b7c977c..66a4b2be8 100644
--- a/output/docs/latest/self-optimizing/index.html
+++ b/output/docs/latest/self-optimizing/index.html
@@ -669,9 +669,9 @@ needs trade off between read and write amplification. Amoro’s Self-optimiz
in the JVM. Files are divided into Fragments and Segments based on their sizes, and different Self-optimizing processes executed on Fragments and
Segments are classified into two types: minor and major. Therefore, Amoro v0.4 introduces two parameters to define Fragments and Segments:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-SQL" data-lang="SQL"><span style="display:flex;"><span><span style="color:#75715e">-- Target file Size for Self-optimizing
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.target<span style="color:#f92672">-</span><span style="color:#66d9ef">size</span> <span style="color:#f92672">=</span> <span style="color:#ae81ff">134217728</span>(<span style="color:#ae81ff">128</span>MB)
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.target<span style="color:#f92672">-</span><span style="color:#66d9ef">size</span> <span style="color:#f92672">=</span> <span style="color:#ae81ff">134217728</span>(<span style="color:#ae81ff">128</span>MB)
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- The fragment file size threshold for Self-optimizing
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.fragment<span style="color:#f92672">-</span>ratio <span style="color:#f92672">=</span> <span style="color:#ae81ff">8</span>
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.fragment<span style="color:#f92672">-</span>ratio <span style="color:#f92672">=</span> <span style="color:#ae81ff">8</span>
</span></span></code></pre></div><p><code>self-optimizing.target-size</code> defines the target output size for major optimizing, which is set to 128 MB by default. <code>self-optimizing.fragment-ratio</code>
defines the ratio of the fragment file threshold to the target-size, with a value of 8 indicating that the default fragment threshold is 1/8 of the
target-size, or 16 MB for a default target-size of 128 MB. Files smaller than 16 MB are considered fragments, while files larger than 16 MB are
@@ -701,38 +701,38 @@ major optimizing.</p>
logic of both optimizing is consistent, as they both involve file compaction, data deduplication, and conversion from write-friendly format to
read-friendly format. The input-output relationships of Minor, Major, and Full optimizing are shown in the following table:</p>
<table>
-<thead>
-<tr>
-<th style="text-align:left">Self-optimizing type</th>
-<th style="text-align:left">Input space</th>
-<th style="text-align:left">Output space</th>
-<th style="text-align:left">Input file types</th>
-<th style="text-align:left">Output file types</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td style="text-align:left">minor</td>
-<td style="text-align:left">fragment</td>
-<td style="text-align:left">fragment/segment</td>
-<td style="text-align:left">insert, eq-delete, pos-delete</td>
-<td style="text-align:left">insert, pos-delete</td>
-</tr>
-<tr>
-<td style="text-align:left">major</td>
-<td style="text-align:left">fragment, segment</td>
-<td style="text-align:left">segment</td>
-<td style="text-align:left">insert, eq-delete, pos-delete</td>
-<td style="text-align:left">insert, pos-delete</td>
-</tr>
-<tr>
-<td style="text-align:left">full</td>
-<td style="text-align:left">fragment, segment</td>
-<td style="text-align:left">segment</td>
-<td style="text-align:left">insert, eq-delete, pos-delete</td>
-<td style="text-align:left">insert</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left">Self-optimizing type</th>
+ <th style="text-align: left">Input space</th>
+ <th style="text-align: left">Output space</th>
+ <th style="text-align: left">Input file types</th>
+ <th style="text-align: left">Output file types</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left">minor</td>
+ <td style="text-align: left">fragment</td>
+ <td style="text-align: left">fragment/segment</td>
+ <td style="text-align: left">insert, eq-delete, pos-delete</td>
+ <td style="text-align: left">insert, pos-delete</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">major</td>
+ <td style="text-align: left">fragment, segment</td>
+ <td style="text-align: left">segment</td>
+ <td style="text-align: left">insert, eq-delete, pos-delete</td>
+ <td style="text-align: left">insert, pos-delete</td>
+ </tr>
+ <tr>
+ <td style="text-align: left">full</td>
+ <td style="text-align: left">fragment, segment</td>
+ <td style="text-align: left">segment</td>
+ <td style="text-align: left">insert, eq-delete, pos-delete</td>
+ <td style="text-align: left">insert</td>
+ </tr>
+ </tbody>
</table>
<h2 id="self-optimizing-scheduling-policy">Self-optimizing scheduling policy</h2>
<p>AMS determines the Scheduling Policy to sequence the self-optimization process for different tables. The actual resources allocated for
@@ -755,10 +755,10 @@ dimensionally aggregated tables), it is advisable to enable the Self-optimizing.
by configuring the quota configuration on the table. Quota specifies the maximum number of optimizer resources that can be allocated to each table.
Quotas can be specified as either a decimal (representing a percentage) or an integer (representing a fixed number of resources):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-SQL" data-lang="SQL"><span style="display:flex;"><span><span style="color:#75715e">-- Set quota as a percentage of total optimizer resources
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.quota <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>.<span style="color:#ae81ff">5</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.quota <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>.<span style="color:#ae81ff">5</span>;
</span></span></code></pre></div><p>A decimal quota (e.g., 0.5) limits the table to a percentage of the total available optimizer resources.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-SQL" data-lang="SQL"><span style="display:flex;"><span><span style="color:#75715e">-- Set quota as a fixed number of optimizer resources
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.quota <span style="color:#f92672">=</span> <span style="color:#ae81ff">10</span>;
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">self</span><span style="color:#f92672">-</span>optimizing.quota <span style="color:#f92672">=</span> <span style="color:#ae81ff">10</span>;
</span></span></code></pre></div><p>An integer quota (e.g., 10) restricts the table to a specific number of optimizer resources.</p>
<p>This flexible configuration prevents resource underutilization and allows users to tailor resource allocation to their needs.</p>
<p>The <code>Quota</code> strategy schedules tables based on their Occupation metric, which is calculated as the ratio of the actual optimizer thread execution time used
diff --git a/output/docs/latest/spark-ddl/index.html b/output/docs/latest/spark-ddl/index.html
index dda2d43ef..6c1de1393 100644
--- a/output/docs/latest/spark-ddl/index.html
+++ b/output/docs/latest/spark-ddl/index.html
@@ -767,25 +767,25 @@ AS SELECT ...
</span></span><span style="display:flex;"><span> );
</span></span></code></pre></div><p>You can add multiple columns at once, separated by commas.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- create a struct column
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> point struct<span style="color:#f92672"><</span>x: double, y: double<span style="color:#f92672">></span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- add a field to the struct
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> point.z double;
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- create a nested array column of struct
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> points array<span style="color:#f92672"><</span>struct<span style="color:#f92672"><</span>x: double, y: double<span style="color:#f92672">>></span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- add a field to the struct within an array. Using keyword 'element' to access the array's element column.
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> points.element.z double;
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#75715e">-- create a map column of struct key and struct value
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> points <span style="color:#66d9ef">map</span><span style="color:#f92672"><</span>struct<span style="color:#f92672"><</span>x: int<span style="color:#f92672">></span>, struct<span style="color:#f92672"><</span>a: int<span style="color:#f92672">>></span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- add a field to the value struct in a map. Using keyword 'value' to access the map's value column.
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ADD</span> <span style="color:#66d9ef">COLUMN</span> points.value.b int;
</span></span></code></pre></div><p>You can add columns at any position by using the <code>FIRST</code> or <code>AFTER</code> clause.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> mixed_catalog.db.sample
diff --git a/output/docs/latest/spark-queries/index.html b/output/docs/latest/spark-queries/index.html
index a548d3e33..3b8fb5085 100644
--- a/output/docs/latest/spark-queries/index.html
+++ b/output/docs/latest/spark-queries/index.html
@@ -641,9 +641,9 @@
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> mixed_catalog.db.sample.change
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">+</span><span style="color:#75715e">---+----+----+---------------+------------+--------------+
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#f92672">|</span> id<span style="color:#f92672">|</span>name<span style="color:#f92672">|</span><span style="color:#66d9ef">data</span><span style="color:#f92672">|</span>_transaction_id<span style="color:#f92672">|</span>_file_offset<span style="color:#f92672">|</span>_change_action<span style="color:#f92672">|</span>
+</span></span></span><span style="display:flex;"><span><span style="color:#f92672">|</span> id<span style="color:#f92672">|</span>name<span style="color:#f92672">|</span><span style="color:#66d9ef">data</span><span style="color:#f92672">|</span>_transaction_id<span style="color:#f92672">|</span>_file_offset<span style="color:#f92672">|</span>_change_action<span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">+</span><span style="color:#75715e">---+----+----+---------------+------------+--------------+
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#f92672">|</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">|</span>dddd<span style="color:#f92672">|</span>abcd<span style="color:#f92672">|</span> <span style="color:#ae81ff">3</span><span style="color:#f92672">|</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">|</span> <span style="color:#66d9ef">INSERT< [...]
+</span></span></span><span style="display:flex;"><span><span style="color:#f92672">|</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">|</span>dddd<span style="color:#f92672">|</span>abcd<span style="color:#f92672">|</span> <span style="color:#ae81ff">3</span><span style="color:#f92672">|</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">|</span> <span style="color:#66d9ef">INSERT</span><span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">|</span> <span style="color:#ae81ff">1</span><span style="color:#f92672">|</span>dddd<span style="color:#f92672">|</span>abcd<span style="color:#f92672">|</span> <span style="color:#ae81ff">3</span><span style="color:#f92672">|</span> <span style="color:#ae81ff">2</span><span style="color:#f92672">|</span> <span style="color:#66d9ef">DELETE</span><span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">+</span><span style="color:#75715e">---+----+----+---------------+------------+--------------+
</span></span></span></code></pre></div><p>The addition columns are:</p>
diff --git a/output/docs/latest/spark-writes/index.html b/output/docs/latest/spark-writes/index.html
index db29f447a..f897b2afe 100644
--- a/output/docs/latest/spark-writes/index.html
+++ b/output/docs/latest/spark-writes/index.html
@@ -713,9 +713,9 @@ an error will be thrown during the write operation.</p>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">WHERE</span> <span style="color:#66d9ef">EXISTS</span> (<span style="color:#66d9ef">SELECT</span> oid <span style="color:#66d9ef">FROM</span> prod.db.returned_orders <span style="color:#66d9ef">WHERE</span> t1.oid <span style="color:#f92672">=</span> oid)
</span></span></code></pre></div><h3 id="merge-into">MERGE INTO</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span>MERGE <span style="color:#66d9ef">INTO</span> prod.db.target t <span style="color:#75715e">-- a target table
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">USING</span> (<span style="color:#66d9ef">SELECT</span> ...) s <span style="color:#75715e">-- the source updates
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ON</span> t.id <span style="color:#f92672">=</span> s.id <span style="color:#75715e">-- condition to find updates for target rows
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">WHEN</span> ... <span style="color:#75715e">-- updates
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">USING</span> (<span style="color:#66d9ef">SELECT</span> ...) s <span style="color:#75715e">-- the source updates
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ON</span> t.id <span style="color:#f92672">=</span> s.id <span style="color:#75715e">-- condition to find updates for target rows
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">WHEN</span> ... <span style="color:#75715e">-- updates
</span></span></span></code></pre></div><p>The <code>MERGE INTO</code> statement supports multi action <code>WHEN MATCHED ... THEN ...</code> to execute <code>UPDATE</code>, <code>DELETE</code>, <code>INSERT</code>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>MERGE <span style="color:#66d9ef">INTO</span> prod.db.target t
diff --git a/output/docs/latest/spark/configuration/index.html b/output/docs/latest/spark/configuration/index.html
index d0391bf9f..8ff26dc25 100644
--- a/output/docs/latest/spark/configuration/index.html
+++ b/output/docs/latest/spark/configuration/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/spark-configuration/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/spark-configuration/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/spark-configuration/">
</head>
diff --git a/output/docs/latest/spark/ddl/index.html b/output/docs/latest/spark/ddl/index.html
index 55930ec16..27c275352 100644
--- a/output/docs/latest/spark/ddl/index.html
+++ b/output/docs/latest/spark/ddl/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/spark-ddl/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/spark-ddl/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/spark-ddl/">
</head>
diff --git a/output/docs/latest/spark/getting-started/index.html b/output/docs/latest/spark/getting-started/index.html
index f70206933..c9ae1efe6 100644
--- a/output/docs/latest/spark/getting-started/index.html
+++ b/output/docs/latest/spark/getting-started/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/spark-getting-started/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/spark-getting-started/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/spark-getting-started/">
</head>
diff --git a/output/docs/latest/spark/queries/index.html b/output/docs/latest/spark/queries/index.html
index f5acfcc25..568b18d4d 100644
--- a/output/docs/latest/spark/queries/index.html
+++ b/output/docs/latest/spark/queries/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/spark-queries/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/spark-queries/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/spark-queries/">
</head>
diff --git a/output/docs/latest/spark/writes/index.html b/output/docs/latest/spark/writes/index.html
index 14ff77866..0baf91772 100644
--- a/output/docs/latest/spark/writes/index.html
+++ b/output/docs/latest/spark/writes/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/spark-writes/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/spark-writes/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/spark-writes/">
</head>
diff --git a/output/docs/latest/trino/index.html b/output/docs/latest/trino/index.html
index b9e65c72e..4686e8637 100644
--- a/output/docs/latest/trino/index.html
+++ b/output/docs/latest/trino/index.html
@@ -670,78 +670,78 @@ The change records of the table can be queried through the ChangeStore, and the
</ul>
<h4 id="trino-and-amoro-type-mapping">Trino and Amoro Type Mapping:</h4>
<table>
-<thead>
-<tr>
-<th style="text-align:left">Amoro type</th>
-<th style="text-align:left">Trino type</th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td style="text-align:left"><code>BOOLEAN</code></td>
-<td style="text-align:left"><code>BOOLEAN</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>INT</code></td>
-<td style="text-align:left"><code>INTEGER</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>LONG</code></td>
-<td style="text-align:left"><code>BIGINT</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>FLOAT</code></td>
-<td style="text-align:left"><code>REAL</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>DOUBLE</code></td>
-<td style="text-align:left"><code>DOUBLE</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>DECIMAL(p,s)</code></td>
-<td style="text-align:left"><code>DECIMAL(p,s)</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>DATE</code></td>
-<td style="text-align:left"><code>DATE</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>TIME</code></td>
-<td style="text-align:left"><code>TIME(6)</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>TIMESTAMP</code></td>
-<td style="text-align:left"><code>TIMESTAMP(6)</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>TIMESTAMPTZ</code></td>
-<td style="text-align:left"><code>TIMESTAMP(6) WITH TIME ZONE</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>STRING</code></td>
-<td style="text-align:left"><code>VARCHAR</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>UUID</code></td>
-<td style="text-align:left"><code>UUID</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>BINARY</code></td>
-<td style="text-align:left"><code>VARBINARY</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>STRUCT(...)</code></td>
-<td style="text-align:left"><code>ROW(...)</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>LIST(e)</code></td>
-<td style="text-align:left"><code>ARRAY(e)</code></td>
-</tr>
-<tr>
-<td style="text-align:left"><code>MAP(k,v)</code></td>
-<td style="text-align:left"><code>MAP(k,v)</code></td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th style="text-align: left">Amoro type</th>
+ <th style="text-align: left">Trino type</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td style="text-align: left"><code>BOOLEAN</code></td>
+ <td style="text-align: left"><code>BOOLEAN</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>INT</code></td>
+ <td style="text-align: left"><code>INTEGER</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>LONG</code></td>
+ <td style="text-align: left"><code>BIGINT</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>FLOAT</code></td>
+ <td style="text-align: left"><code>REAL</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>DOUBLE</code></td>
+ <td style="text-align: left"><code>DOUBLE</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>DECIMAL(p,s)</code></td>
+ <td style="text-align: left"><code>DECIMAL(p,s)</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>DATE</code></td>
+ <td style="text-align: left"><code>DATE</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>TIME</code></td>
+ <td style="text-align: left"><code>TIME(6)</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>TIMESTAMP</code></td>
+ <td style="text-align: left"><code>TIMESTAMP(6)</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>TIMESTAMPTZ</code></td>
+ <td style="text-align: left"><code>TIMESTAMP(6) WITH TIME ZONE</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>STRING</code></td>
+ <td style="text-align: left"><code>VARCHAR</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>UUID</code></td>
+ <td style="text-align: left"><code>UUID</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>BINARY</code></td>
+ <td style="text-align: left"><code>VARBINARY</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>STRUCT(...)</code></td>
+ <td style="text-align: left"><code>ROW(...)</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>LIST(e)</code></td>
+ <td style="text-align: left"><code>ARRAY(e)</code></td>
+ </tr>
+ <tr>
+ <td style="text-align: left"><code>MAP(k,v)</code></td>
+ <td style="text-align: left"><code>MAP(k,v)</code></td>
+ </tr>
+ </tbody>
</table>
<h3 id="trino-uses-proxy-user-to-access-hadoop-cluster">Trino uses proxy user to access Hadoop cluster.</h3>
<p>By default, when Trino queries Amoro, it uses the Hadoop user configured in the <a href="../managing-catalogs/#create-catalog">catalog creation</a> to access the Hadoop cluster.
diff --git a/output/docs/latest/user-guides/cdc-ingestion/index.html b/output/docs/latest/user-guides/cdc-ingestion/index.html
index a5e76d5b1..b4481305a 100644
--- a/output/docs/latest/user-guides/cdc-ingestion/index.html
+++ b/output/docs/latest/user-guides/cdc-ingestion/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/cdc-ingestion/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/cdc-ingestion/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/cdc-ingestion/">
</head>
diff --git a/output/docs/latest/user-guides/configurations/index.html b/output/docs/latest/user-guides/configurations/index.html
index a3d22f270..d06fc6b2d 100644
--- a/output/docs/latest/user-guides/configurations/index.html
+++ b/output/docs/latest/user-guides/configurations/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/configurations/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/configurations/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/configurations/">
</head>
diff --git a/output/docs/latest/user-guides/index.xml b/output/docs/latest/user-guides/index.xml
index 2e4669198..7b73cef2a 100644
--- a/output/docs/latest/user-guides/index.xml
+++ b/output/docs/latest/user-guides/index.xml
@@ -12,35 +12,35 @@
<link>https://amoro.apache.org/docs/latest/cdc-ingestion/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/cdc-ingestion/</guid>
- <description>CDC Ingestion CDC stands for Change Data Capture, which is a broad concept, as long as it can capture the change data, it can be called CDC. Flink CDC is a Log message-based data capture tool, all the inventory and incremental data can be captured. Taking MySQL as an example, it can easily capture Binlog data through Debezium、Flink CDC and process the calculations in real time to send them to the data lake.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Configurations</title>
<link>https://amoro.apache.org/docs/latest/configurations/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/configurations/</guid>
- <description>Table Configurations Multi-level configuration management Amoro provides configurations that can be configured at the Catalog, Table, and Engine levels. The configuration priority is given first to the Engine, followed by the Table, and finally by the Catalog.
Catalog: Generally, we recommend users to set default values for tables through the Catalog properties configuration, such as Self-optimizing related configurations. Table: We also recommend users to specify [...]
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Metrics</title>
<link>https://amoro.apache.org/docs/latest/metrics/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/metrics/</guid>
- <description>Metrics Amoro build a metrics system to measure the behaviours of table management processes, like how long has it been since a table last performed self-optimizing process, and how much resources does a optimizer group currently has?
There are two types of metrics provided in the Amoro metric system: Gauge and Counter.
Gauge: Provides a value of any type at a point in time. Counter: Used to count values by incrementing and decrementing.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>REST API</title>
<link>https://amoro.apache.org/docs/latest/rest-api/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/rest-api/</guid>
- <description> REST API Accessing Swagger UI Access the Swagger UI at:
http://&lt;host&gt;:&lt;port&gt;/#/openapi-ui You can also view the OpenAPI specification YAML in the repository at amoro-ams/src/main/resources/openapi/openapi.yaml, or use the Swagger Editor online.
Building the OpenAPI SDK Build the OpenAPI SDK using Maven:
./mvnw clean package -pl amoro-openapi-sdk -am -Popenapi-sdk </description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
<item>
<title>Using Tables</title>
<link>https://amoro.apache.org/docs/latest/using-tables/</link>
<pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
<guid>https://amoro.apache.org/docs/latest/using-tables/</guid>
- <description>Using Tables The SQL execution tool Terminal is provided in AMS dashboard to help users quickly create, modify and delete tables. It is also available in Spark and Flink and other engines to manage tables using SQL.
Create table After logging into AMS dashboard, go to Terminal, enter the table creation statement and execute it to complete the table creation. The following is an example of table creation:
create table test_db.</description>
+ <description><!--
 - Licensed to the Apache Software Foundation (ASF) under one or more
 - contributor license agreements. See the NOTICE file distributed with
 - this work for additional information regarding copyright ownership.
 - The ASF licenses this file to You under the Apache License, Version 2.0
 - (the "License"); you may not use this file except in compliance with
 - the License. You may obtain a copy of the License at
 -
 - [...]
</item>
</channel>
</rss>
diff --git a/output/docs/latest/user-guides/metrics/index.html b/output/docs/latest/user-guides/metrics/index.html
index 225b1fe8e..05115c1c5 100644
--- a/output/docs/latest/user-guides/metrics/index.html
+++ b/output/docs/latest/user-guides/metrics/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/metrics/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/metrics/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/metrics/">
</head>
diff --git a/output/docs/latest/user-guides/rest-api/index.html b/output/docs/latest/user-guides/rest-api/index.html
index ae4285050..f3e30084b 100644
--- a/output/docs/latest/user-guides/rest-api/index.html
+++ b/output/docs/latest/user-guides/rest-api/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/rest-api/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/rest-api/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/rest-api/">
</head>
diff --git a/output/docs/latest/user-guides/using-tables/index.html b/output/docs/latest/user-guides/using-tables/index.html
index 69e1a5138..dd3e3b139 100644
--- a/output/docs/latest/user-guides/using-tables/index.html
+++ b/output/docs/latest/user-guides/using-tables/index.html
@@ -3,7 +3,6 @@
<head>
<title>https://amoro.apache.org/docs/latest/using-tables/</title>
<link rel="canonical" href="https://amoro.apache.org/docs/latest/using-tables/">
- <meta name="robots" content="noindex">
<meta charset="utf-8">
<meta http-equiv="refresh" content="0; url=https://amoro.apache.org/docs/latest/using-tables/">
</head>
diff --git a/output/docs/latest/using-tables/index.html b/output/docs/latest/using-tables/index.html
index e82660836..b586c508b 100644
--- a/output/docs/latest/using-tables/index.html
+++ b/output/docs/latest/using-tables/index.html
@@ -729,12 +729,12 @@ Note that timestamp, timestampz, and long field types use UTC, while others use
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">'data-expire.field'</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">'op_time'</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- select string field
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> test_db.test_log_store <span style="color:#66d9ef">set</span> tblproperties (
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> test_db.test_log_store <span style="color:#66d9ef">set</span> tblproperties (
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">'data-expire.field'</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">'op_time'</span>,
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">'data-expire.datetime-string-pattern'</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">'yyyy-MM-dd'</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">-- select long field
-</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> test_db.test_log_store <span style="color:#66d9ef">set</span> tblproperties (
+</span></span></span><span style="display:flex;"><span><span style="color:#66d9ef">ALTER</span> <span style="color:#66d9ef">TABLE</span> test_db.test_log_store <span style="color:#66d9ef">set</span> tblproperties (
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">'data-expire.field'</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">'op_time'</span>,
</span></span><span style="display:flex;"><span> <span style="color:#e6db74">'data-expire.datetime-number-format'</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">'TIMESTAMP_MS'</span>);
</span></span></code></pre></div><h3 id="adjust-expiration-level">Adjust expiration level</h3>
@@ -753,34 +753,34 @@ Note that timestamp, timestampz, and long field types use UTC, while others use
<h2 id="explore-table-details">Explore table details</h2>
<p>The Amoro Tables details page provides multiple tabs to display the status of the table from various dimensions, mainly including:</p>
<table>
-<thead>
-<tr>
-<th><strong>Tab Name</strong></th>
-<th><strong>Description</strong></th>
-</tr>
-</thead>
-<tbody>
-<tr>
-<td>Details</td>
-<td>Display the table’s schema, primary key configuration, partition configuration, properties; as well as the metric information of the files stored in ChangeStore and BaseStore, including the number of files and average file size, as well as the latest submission time of the files.</td>
-</tr>
-<tr>
-<td>Files</td>
-<td>Display all partitions and files of the table.</td>
-</tr>
-<tr>
-<td>Snapshots</td>
-<td>Display all snapshots of the table, which can be filtered by branch and tag.</td>
-</tr>
-<tr>
-<td>Optimizing</td>
-<td>Display all the self-optimizing processes of the table, each record shows the number and average size of files before and after Optimize, as well as the execution time of each process.</td>
-</tr>
-<tr>
-<td>Operations</td>
-<td>Display the current table’s DDL historical change records.</td>
-</tr>
-</tbody>
+ <thead>
+ <tr>
+ <th><strong>Tab Name</strong></th>
+ <th><strong>Description</strong></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Details</td>
+ <td>Display the table’s schema, primary key configuration, partition configuration, properties; as well as the metric information of the files stored in ChangeStore and BaseStore, including the number of files and average file size, as well as the latest submission time of the files.</td>
+ </tr>
+ <tr>
+ <td>Files</td>
+ <td>Display all partitions and files of the table.</td>
+ </tr>
+ <tr>
+ <td>Snapshots</td>
+ <td>Display all snapshots of the table, which can be filtered by branch and tag.</td>
+ </tr>
+ <tr>
+ <td>Optimizing</td>
+ <td>Display all the self-optimizing processes of the table, each record shows the number and average size of files before and after Optimize, as well as the execution time of each process.</td>
+ </tr>
+ <tr>
+ <td>Operations</td>
+ <td>Display the current table’s DDL historical change records.</td>
+ </tr>
+ </tbody>
</table>
<p><img src="../images/admin/table_metrics.png" alt="table-details"></p>
<p><img src="../images/admin/table_optimizer_history.png" alt="table-optimize-history"></p>