You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by cz...@apache.org on 2026/07/06 03:29:12 UTC
(amoro) 05/10: [JSPT-2570] Stp-Iceberg的 Producer入口统一迁移至amoro-producer模块并做统一优化
This is an automated email from the ASF dual-hosted git repository.
czy006 pushed a commit to branch jira/JSPT-2568
in repository https://gitbox.apache.org/repos/asf/amoro.git
commit 0a686b738e864aac76b35ca27b2959fecca562db
Author: ConradJam <ja...@gmail.com>
AuthorDate: Tue Jun 30 10:25:23 2026 +0800
[JSPT-2570] Stp-Iceberg的 Producer入口统一迁移至amoro-producer模块并做统一优化
---
amoro-producer/README.md | 914 ++++++++++++---------
amoro-producer/pom.xml | 2 +-
.../apache/amoro/producer/LakehouseProducer.scala | 32 +
.../amoro/producer/PaimonCompactManifest.scala | 424 ----------
.../amoro/producer/PaimonExpireSnapshots.scala | 491 -----------
.../amoro/producer/PaimonRemoveOrphanFiles.scala | 478 -----------
.../apache/amoro/producer/core/ActionResult.scala | 40 +
.../apache/amoro/producer/core/ProducerApp.scala | 307 +++++++
.../amoro/producer/core/ProducerCliParser.scala | 243 ++++++
.../amoro/producer/core/ProducerConfig.scala | 29 +
.../amoro/producer/core/ProducerContext.scala | 28 +
.../amoro/producer/core/ProducerOptions.scala | 51 ++
.../amoro/producer/core/ProducerSummary.scala | 76 ++
.../org/apache/amoro/producer/core/RetryMode.scala | 26 +
.../amoro/producer/core/TableActionJsonLog.scala | 125 +++
.../apache/amoro/producer/core/TableFilter.scala | 53 ++
.../apache/amoro/producer/core/TableResolver.scala | 109 +++
.../apache/amoro/producer/core/TargetTable.scala | 21 +
.../producer/paimon/PaimonCompactAction.scala | 373 +++++++++
.../paimon/PaimonCompactManifestAction.scala | 125 +++
.../paimon/PaimonExpireSnapshotsAction.scala | 260 ++++++
.../producer/paimon/PaimonFormatAdapter.scala | 79 ++
.../paimon/PaimonRemoveOrphanFilesAction.scala | 159 ++++
.../amoro/producer/paimon/PaimonSqlRunner.scala | 33 +
.../amoro/producer/paimon/PaimonSqlUtils.scala | 63 ++
.../amoro/producer/spi/LakeFormatAdapter.scala | 35 +
.../apache/amoro/producer/spi/ProducerAction.scala | 41 +
.../amoro/producer/LakehouseProducerTest.scala | 398 +++++++++
.../amoro/producer/core/ActionResultTest.scala | 129 +++
.../amoro/producer/core/ProducerAppTest.scala | 584 +++++++++++++
.../producer/core/ProducerCliParserTest.scala | 246 ++++++
.../amoro/producer/core/TableFilterTest.scala | 52 ++
.../amoro/producer/core/TableResolverTest.scala | 200 +++++
.../producer/paimon/PaimonCompactActionTest.scala | 395 +++++++++
.../paimon/PaimonCompactManifestActionTest.scala | 167 ++++
.../paimon/PaimonExpireSnapshotsActionTest.scala | 320 ++++++++
.../producer/paimon/PaimonFormatAdapterTest.scala | 174 ++++
.../paimon/PaimonRemoveOrphanFilesActionTest.scala | 175 ++++
38 files changed, 5666 insertions(+), 1791 deletions(-)
diff --git a/amoro-producer/README.md b/amoro-producer/README.md
index d92ba1b05..d5b76a994 100644
--- a/amoro-producer/README.md
+++ b/amoro-producer/README.md
@@ -15,13 +15,20 @@
- limitations under the License.
-->
-# Amoro Producer Paimon 清理任务
+# Amoro Lakehouse Producer
-`amoro-producer` 提供可通过 `spark-submit` 提交的 Paimon 清理任务:
+`amoro-producer` 提供统一的 Lakehouse Producer 入口:
+`org.apache.amoro.producer.LakehouseProducer`。
-- Snapshot 清理入口类为 `org.apache.amoro.producer.PaimonExpireSnapshots`,任务会通过 Spark SQL 执行 `CALL sys.expire_snapshots`。
-- 孤儿文件清理入口类为 `org.apache.amoro.producer.PaimonRemoveOrphanFiles`,任务会通过 Spark SQL 执行 `CALL <catalog>.sys.remove_orphan_files`。
-- Manifest 合并入口类为 `org.apache.amoro.producer.PaimonCompactManifest`,任务会通过 Spark SQL 执行 `CALL <catalog>.sys.compact_manifest`。
+当前支持 Paimon format,提交任务时必须指定:
+
+- `--format paimon`
+- `--action <action>`
+- `--catalogName paimon`
+
+`--catalogName` 的默认值是 `spark_catalog`。Paimon 生产脚本必须显式传入
+`--catalogName paimon`,该参数用于整库扫描、`$options` 读取和 catalog-qualified
+procedure 调用。
## 打包
@@ -35,449 +42,562 @@
amoro-producer/target/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar
```
-## Snapshot 清理参数说明
+## 通用参数
| 参数 | 是否必填 | 默认值 | 说明 |
|------|----------|--------|------|
-| `--databaseName` | 否 | 无 | 数据库名。仅传该参数时,任务会扫描并清理该库下全部非临时表。 |
-| `--tableName` | 否 | 无 | 表名。与 `--databaseName` 同时传入时,只清理指定表;单独传入时必须使用 `db.table` 格式。 |
-| `--retainMax` | 否 | `20` | 传给 `expire_snapshots` 的 `retain_max`。 |
-| `--retainMin` | 否 | `10` | 传给 `expire_snapshots` 的 `retain_min`。 |
-| `--maxDeletes` | 否 | `550` | 单轮最多删除的 snapshot 数量。若返回删除数大于等于该值,则继续下一轮清理。 |
-| `--retryTimes` | 否 | `3` | 单轮 `expire_snapshots` SQL 失败后的最大重试次数。 |
-| `--continueOnTableFailure` | 否 | `true` | 单表清理失败后是否继续处理后续表。 |
-| `--procedureOptions` | 否 | `file-operation.thread-num=32,snapshot.expire.execution-mode=sync` | 传给 `expire_snapshots` 的 `options` 字符串。传空字符串时不拼接 `options` 参数。 |
+| `--format` | 是 | 无 | 表格式。Paimon 使用 `paimon`。 |
+| `--action` | 是 | 无 | action 名称:`compact`、`compact-manifest`、`expire-snapshots`、`remove-orphan-files`。 |
+| `--catalogName` | 否 | `spark_catalog` | Spark catalog 名称。Paimon 生产脚本应显式传 `paimon`。 |
+| `--databaseName` | 否 | 无 | 数据库名。仅传该参数时,扫描该库下全部非临时表。 |
+| `--tableName` | 否 | 无 | 表名。与 `--databaseName` 同时传入时只允许简单表名;单独传入时必须是 `db.table`。 |
+| `--tableNameRegex` | 否 | 无 | 整库模式表名过滤正则,仅支持 `--databaseName` 且未指定 `--tableName`。 |
+| `--retryTimes` | 否 | `3` | 重试次数。 |
+| `--continueOnTableFailure` | 否 | `true` | 单表失败后是否继续处理后续表。 |
| `--help` / `-h` | 否 | 无 | 打印帮助信息。 |
-参数组合规则:
-
-| 参数组合 | 行为 |
-|----------|------|
-| `--databaseName db` | 清理 `db` 下所有非临时表。 |
-| `--databaseName db --tableName t1` | 只清理 `db.t1`。 |
-| `--tableName db.t1` | 只清理 `db.t1`。 |
-| `--tableName t1` | 非法参数,任务打印 usage 并以退出码 `2` 结束。 |
-| `--databaseName db --tableName db.t1` | 非法参数,任务打印 usage 并以退出码 `2` 结束。 |
+## compact
-## 孤儿文件清理参数说明
+执行 Paimon `compact` procedure。
| 参数 | 是否必填 | 默认值 | 说明 |
|------|----------|--------|------|
-| `--catalogName` | 否 | `paimon` | Paimon catalog 名称。任务会使用该 catalog 扫表并调用 `remove_orphan_files`。 |
-| `--databaseName` | 否 | 无 | 数据库名。仅传该参数时,任务会扫描并清理该库下全部非临时表。 |
-| `--tableName` | 否 | 无 | 表名。与 `--databaseName` 同时传入时,只清理指定表;单独传入时必须使用 `db.table` 格式。 |
-| `--parallelism` | 否 | `10` | 传给 `remove_orphan_files` 的 `parallelism`。 |
-| `--mode` | 否 | `distributed` | 传给 `remove_orphan_files` 的 `mode`。 |
-| `--retryTimes` | 否 | `3` | 单表 `remove_orphan_files` SQL 失败后的最大执行次数。 |
-| `--continueOnTableFailure` | 否 | `true` | 单表清理失败后是否继续处理后续表。 |
-| `--help` / `-h` | 否 | 无 | 打印帮助信息。 |
+| `--startBucket` | 否 | `0` | 从哪个 bucket 下标开始执行。 |
+| `--step` | 否 | `20` | 每次 compact 的 bucket 数量,生成闭区间 bucket range,例如 `0-19`。 |
+| `--compactStrategy` | 否 | `full` | 传给 `compact` 的 `compact_strategy`。 |
+| `--procedureOptions` | 否 | `target-file-size=256m` | 传给 `compact` 的 `options` 字符串,空字符串表示不拼接。 |
+| `--version` | 否 | 空字符串 | 设置为 `0.9` 时使用 Paimon 0.9 `partition_idle_time` SQL。 |
+| `--partitionIdleTime` | 否 | `1d` | `--version 0.9` 时传给 `compact` 的 `partition_idle_time`。 |
+
+示例:
+
+```bash
+./spark-submit \
+ --master yarn \
+ --deploy-mode cluster \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.defaultCatalog=paimon \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=hdfs://warehouse/paimon \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
+ --format paimon \
+ --action compact \
+ --catalogName paimon \
+ --databaseName ods \
+ --tableNameRegex '^ods_.*' \
+ --startBucket 0 \
+ --step 20 \
+ --compactStrategy full \
+ --procedureOptions target-file-size=256m
+```
+
+compact 会先读取目标表 `$options` 中的 `bucket` 配置:
-孤儿文件清理的目标表参数组合规则与 Snapshot 清理保持一致。
+- 有效正整数 bucket 且未设置 `--version 0.9` 时,按 `--startBucket` 和 `--step`
+ 生成 bucket range,并调用带 `buckets`、`compact_strategy`、`options` 的
+ `compact` SQL。
+- bucket 缺失、非整数、`<= 0` 或读取失败时,无论 version 如何,都执行
+ `CALL sys.compact(table => 'db.table')`。
+- 设置 `--version 0.9` 且读取到有效 bucket 时,执行带 `partition_idle_time`
+ 的 SQL,不拼接 `buckets`、`compact_strategy` 或 `options`。
-## Manifest 合并参数说明
+其中非 bucket compact 和 `--version 0.9` bucket compact 使用
+`CALL sys.compact(...)`,依赖 Spark 的 `spark.sql.defaultCatalog` 指向 Paimon catalog。
+生产提交时应设置 `--conf spark.sql.defaultCatalog=paimon`,并与 `--catalogName paimon`
+保持一致。
+
+## compact-manifest
+
+执行 Paimon `compact_manifest` procedure。
| 参数 | 是否必填 | 默认值 | 说明 |
|------|----------|--------|------|
-| `--catalogName` | 否 | `paimon` | Paimon catalog 名称。任务会使用该 catalog 扫表并调用 `compact_manifest`。 |
-| `--databaseName` | 否 | 无 | 数据库名。仅传该参数时,任务会扫描并处理该库下全部非临时表。 |
-| `--tableName` | 否 | 无 | 表名。与 `--databaseName` 同时传入时,只处理指定表;单独传入时必须使用 `db.table` 格式。 |
-| `--procedureOptions` | 否 | 空字符串 | 传给 `compact_manifest` 的 `options` 字符串。传空字符串时不拼接 `options` 参数。 |
-| `--retryTimes` | 否 | `3` | 单表 `compact_manifest` SQL 失败后的最大执行次数。 |
-| `--continueOnTableFailure` | 否 | `true` | 单表 Manifest 合并失败后是否继续处理后续表。 |
-| `--help` / `-h` | 否 | 无 | 打印帮助信息。 |
+| `--procedureOptions` | 否 | 空字符串 | 传给 `compact_manifest` 的 `options` 字符串,空字符串表示不拼接。 |
+
+示例:
+
+```bash
+./spark-submit \
+ --master yarn \
+ --deploy-mode cluster \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=hdfs://warehouse/paimon \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
+ --format paimon \
+ --action compact-manifest \
+ --catalogName paimon \
+ --databaseName ods \
+ --procedureOptions file-operation.thread-num=16
+```
-Manifest 合并的目标表参数组合规则与 Snapshot 清理保持一致。
+## expire-snapshots
-## Snapshot 清理 Spark Submit Demo
+执行 Paimon `expire_snapshots` procedure。
-以下示例基于 YARN cluster 模式提交,主类替换为
-`org.apache.amoro.producer.PaimonExpireSnapshots`,应用 JAR 替换为
-`amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar`。示例中显式声明
-Paimon catalog,并通过 `spark.sql.defaultCatalog=paimon` 让 `--databaseName`
-按 Paimon catalog 下的库解析。
+| 参数 | 是否必填 | 默认值 | 说明 |
+|------|----------|--------|------|
+| `--retainMax` | 否 | `20` | 传给 `expire_snapshots` 的 `retain_max`。 |
+| `--retainMin` | 否 | `10` | 传给 `expire_snapshots` 的 `retain_min`。 |
+| `--maxDeletes` | 否 | `550` | 单轮最多删除的 snapshot 数量;删除数达到该值会继续下一轮。 |
+| `--procedureOptions` | 否 | `file-operation.thread-num=32,snapshot.expire.execution-mode=sync` | 传给 `expire_snapshots` 的 `options` 字符串,空字符串表示不拼接。 |
-### 按数据库清理
+示例:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonExpireSnapshots \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --databaseName sl_oki_test \
- --retainMax 20 \
- --retainMin 10 \
- --maxDeletes 550 \
- --retryTimes 3 \
- --continueOnTableFailure true \
- --procedureOptions file-operation.thread-num=32,snapshot.expire.execution-mode=sync
+ --master yarn \
+ --deploy-mode cluster \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=hdfs://warehouse/paimon \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
+ --format paimon \
+ --action expire-snapshots \
+ --catalogName paimon \
+ --databaseName ods \
+ --retainMax 20 \
+ --retainMin 10 \
+ --maxDeletes 550 \
+ --procedureOptions file-operation.thread-num=32,snapshot.expire.execution-mode=sync
```
-### Snapshot 按单表清理
+## remove-orphan-files
+
+执行 Paimon `remove_orphan_files` procedure。
+
+| 参数 | 是否必填 | 默认值 | 说明 |
+|------|----------|--------|------|
+| `--parallelism` | 否 | `10` | 传给 `remove_orphan_files` 的 `parallelism`。 |
+| `--mode` | 否 | `distributed` | 传给 `remove_orphan_files` 的 `mode`。 |
+
+示例:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonExpireSnapshots \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --databaseName sl_oki_test \
- --tableName t_order \
- --retainMax 20 \
- --retainMin 10 \
- --maxDeletes 550 \
- --retryTimes 3 \
- --continueOnTableFailure true \
- --procedureOptions file-operation.thread-num=32,snapshot.expire.execution-mode=sync
+ --master yarn \
+ --deploy-mode cluster \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=hdfs://warehouse/paimon \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
+ --format paimon \
+ --action remove-orphan-files \
+ --catalogName paimon \
+ --databaseName ods \
+ --parallelism 10 \
+ --mode distributed
```
-## 孤儿文件清理 Spark Submit Demo
+## 生产 spark-submit 示例
-孤儿文件清理入口显式接收 `--catalogName`,示例中使用 `paimon`。如果生产环境
-catalog 名称不同,只需要调整 `--catalogName` 以及对应的
-`spark.sql.catalog.<catalog>` 配置。
+以下示例以 Paimon catalog `paimon`、数据库 `sl_oki_prod` 为例。`spark.driver.host`、
+`spark.sql.catalog.paimon.warehouse`、`--keytab`、`--principal`、`--jars` 和 producer jar
+路径需要替换为实际生产环境路径。
-### 孤儿文件按数据库清理
+如果目标库表数量很多,建议在整库模式下增加 `--tableNameRegex`,只处理命中正则的表。
+例如 `--tableNameRegex '^orders_.*'` 只处理 `sl_oki_prod` 下表名匹配该正则的表。
+
+### expire-snapshots
+
+整库快照过期:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonRemoveOrphanFiles \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --catalogName paimon \
- --databaseName sl_oki_test \
- --parallelism 10 \
- --mode distributed \
- --retryTimes 3 \
- --continueOnTableFailure true
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action expire-snapshots \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --retainMax 20 \
+ --retainMin 10 \
+ --maxDeletes 550 \
+ --procedureOptions file-operation.thread-num=32,snapshot.expire.execution-mode=sync
```
-### 孤儿文件按单表清理
+只处理命中正则的表:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonRemoveOrphanFiles \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --catalogName paimon \
- --databaseName sl_oki_test \
- --tableName t_order \
- --parallelism 10 \
- --mode distributed \
- --retryTimes 3 \
- --continueOnTableFailure true
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action expire-snapshots \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --tableNameRegex '^orders_.*' \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --retainMax 20 \
+ --retainMin 10 \
+ --maxDeletes 550 \
+ --procedureOptions file-operation.thread-num=32,snapshot.expire.execution-mode=sync
```
-## Manifest 合并 Spark Submit Demo
-
-Manifest 合并入口显式接收 `--catalogName`,示例中使用 `paimon`。如果生产环境
-catalog 名称不同,只需要调整 `--catalogName` 以及对应的
-`spark.sql.catalog.<catalog>` 配置。
+### compact
-### Manifest 按数据库合并
+整库小文件合并:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonCompactManifest \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --catalogName paimon \
- --databaseName sl_oki_test \
- --retryTimes 3 \
- --continueOnTableFailure true
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action compact \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --tableNameRegex '^orders_.*' \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --startBucket 0 \
+ --step 20 \
+ --compactStrategy full \
+ --procedureOptions target-file-size=256m
```
-### Manifest 按单表合并
+Paimon 0.9 bucket 表 compact 使用 `partition_idle_time`:
```bash
./spark-submit \
- --master yarn \
- --deploy-mode cluster \
- --driver-memory 8g \
- --driver-cores 4 \
- --executor-memory 4g \
- --executor-cores 4 \
- --conf spark.dynamicAllocation.enabled=true \
- --conf spark.dynamicAllocation.minExecutors=1 \
- --conf spark.dynamicAllocation.maxExecutors=2 \
- --conf spark.dynamicAllocation.initialExecutors=1 \
- --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
- --conf spark.driver.memoryOverhead=2g \
- --conf spark.executor.memoryOverhead=2g \
- --conf spark.driver.bindAddress=0.0.0.0 \
- --conf spark.sql.shuffle.partitions=400 \
- --conf spark.sql.cbo.enabled=true \
- --conf spark.sql.files.maxPartitionBytes=512m \
- --conf spark.default.parallelism=200 \
- --conf spark.locality.wait=0s \
- --conf spark.sql.crossJoin.enabled=true \
- --conf spark.sql.adaptive.localShuffleReader.enabled=true \
- --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
- --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
- --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
- --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
- --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
- --conf spark.sql.optimizer.groupSplitsByLocation=false \
- --conf spark.sql.adaptive.amend.join.selection.enabled=true \
- --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
- --conf spark.sql.execution.optimizeExpand=true \
- --conf spark.sql.execution.optimizeExpand.ratio=5 \
- --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
- --conf spark.sql.auto.reused.cte.enabled=true \
- --conf spark.sql.auto.clear.cte.cache.enabled=true \
- --conf spark.locality.wait.node=0s \
- --conf spark.sql.sources.ignoreDataLocality=true \
- --conf spark.driver.host=10.89.56.103 \
- --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
- --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
- --conf spark.sql.catalog.paimon.type=hive \
- --conf spark.sql.catalog.paimon.warehouse=hdfs://slcluster01/hive_warehouse \
- --conf spark.hadoop.net.topology.script.file.name=/dev/null \
- --conf spark.sql.defaultCatalog=paimon \
- --keytab /data/work/hadoop-shopline/etc/hadoop/sljdp.keytab \
- --jars /tmp/paimon-hive-connector-3.1-1.5-SNAPSHOT.jar,/tmp/paimon-spark-3.5_2.12-1.5-SNAPSHOT.jar \
- --principal sljdp@SLCLUSTER.COM \
- --class org.apache.amoro.producer.PaimonCompactManifest \
- /data/work/frame/amoro-0.9-SNAPSHOT/plugin/producer/amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar \
- --catalogName paimon \
- --databaseName sl_oki_test \
- --tableName t_order \
- --retryTimes 3 \
- --continueOnTableFailure true
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action compact \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --tableNameRegex '^orders_.*' \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --version 0.9 \
+ --partitionIdleTime 1d
```
-## 运行注意事项
+### compact-manifest
-Spark 运行环境必须提前配置好 Paimon catalog,否则 `SHOW TABLES` 或
-`CALL sys.expire_snapshots`、`CALL <catalog>.sys.remove_orphan_files`、
-`CALL <catalog>.sys.compact_manifest` 会在运行期失败。
+整库 manifest 合并:
-如果运行环境已经通过 Spark 安装目录或平台层提供 Paimon 相关 JAR,可以直接提交
-`amoro-producer-0.9-SNAPSHOT.jar`;如果希望单 JAR 提交,使用
-`amoro-producer-0.9-SNAPSHOT-jar-with-dependencies.jar`。
+```bash
+./spark-submit \
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action compact-manifest \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --tableNameRegex '^orders_.*' \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --procedureOptions file-operation.thread-num=16
+```
-`spark.dynamicAllocation.initialExecutors` 建议不要大于
-`spark.dynamicAllocation.maxExecutors`。上面的示例保留了原始提交命令中的资源参数,实际提交前请按集群策略调整。
+### remove-orphan-files
+
+整库删除孤儿文件:
+
+```bash
+./spark-submit \
+ --master yarn \
+ --deploy-mode cluster \
+ --driver-memory 8g \
+ --driver-cores 4 \
+ --executor-memory 8g \
+ --executor-cores 2 \
+ --conf spark.dynamicAllocation.enabled=true \
+ --conf spark.dynamicAllocation.minExecutors=1 \
+ --conf spark.dynamicAllocation.maxExecutors=2 \
+ --conf spark.dynamicAllocation.initialExecutors=1 \
+ --conf spark.dynamicAllocation.shuffleTracking.enabled=true \
+ --conf spark.yarn.queue=real_time_merge \
+ --conf spark.driver.memoryOverhead=2g \
+ --conf spark.executor.memoryOverhead=2g \
+ --conf spark.driver.bindAddress=0.0.0.0 \
+ --conf spark.sql.shuffle.partitions=400 \
+ --conf spark.sql.cbo.enabled=true \
+ --conf spark.sql.files.maxPartitionBytes=512m \
+ --conf spark.default.parallelism=200 \
+ --conf spark.locality.wait=0s \
+ --conf spark.sql.crossJoin.enabled=true \
+ --conf spark.sql.adaptive.localShuffleReader.enabled=true \
+ --conf spark.sql.fuse.unionAllOnJoin.enabled=true \
+ --conf spark.sql.optimizer.runtime.bloomFilter.enabled=true \
+ --conf spark.sql.optimizer.enableMergeScalarAggsInInnerJoin=true \
+ --conf spark.sql.optimizer.pushdownAggregateBelowJoin=true \
+ --conf spark.sql.optimizer.inferDistinctFromIntersect=true \
+ --conf spark.sql.optimizer.groupSplitsByLocation=false \
+ --conf spark.sql.adaptive.amend.join.selection.enabled=true \
+ --conf spark.sql.mergeScalaSubquery.pullupAggFilter=true \
+ --conf spark.sql.execution.optimizeExpand=true \
+ --conf spark.sql.execution.optimizeExpand.ratio=5 \
+ --conf spark.sql.legacy.ctePrecedencePolicy=LEGACY \
+ --conf spark.sql.auto.reused.cte.enabled=true \
+ --conf spark.sql.auto.clear.cte.cache.enabled=true \
+ --conf spark.locality.wait.node=0s \
+ --conf spark.sql.sources.ignoreDataLocality=true \
+ --conf spark.driver.host=REPLACE_DRIVER_HOST \
+ --conf spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog \
+ --conf spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions \
+ --conf spark.sql.catalog.paimon.type=hive \
+ --conf spark.sql.catalog.paimon.warehouse=REPLACE_PAIMON_WAREHOUSE \
+ --conf spark.hadoop.net.topology.script.file.name=/dev/null \
+ --conf spark.sql.defaultCatalog=paimon \
+ --keytab REPLACE_KEYTAB_FILE \
+ --jars REPLACE_PAIMON_HIVE_CONNECTOR_JAR,REPLACE_PAIMON_SPARK_JAR \
+ --principal REPLACE_PRINCIPAL \
+ --class org.apache.amoro.producer.LakehouseProducer \
+ REPLACE_AMORO_PRODUCER_JAR \
+ --format paimon \
+ --action remove-orphan-files \
+ --catalogName paimon \
+ --databaseName sl_oki_prod \
+ --tableNameRegex '^orders_.*' \
+ --retryTimes 3 \
+ --continueOnTableFailure true \
+ --parallelism 10 \
+ --mode distributed
+```
diff --git a/amoro-producer/pom.xml b/amoro-producer/pom.xml
index e590ebffc..7fbf69409 100644
--- a/amoro-producer/pom.xml
+++ b/amoro-producer/pom.xml
@@ -123,7 +123,7 @@
</artifactSet>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
- <mainClass>org.apache.amoro.producer.PaimonExpireSnapshots</mainClass>
+ <mainClass>org.apache.amoro.producer.LakehouseProducer</mainClass>
</transformer>
</transformers>
</configuration>
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/LakehouseProducer.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/LakehouseProducer.scala
new file mode 100644
index 000000000..5941d63ce
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/LakehouseProducer.scala
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer
+
+import org.apache.amoro.producer.core.ProducerApp
+import org.apache.amoro.producer.paimon.PaimonFormatAdapter
+import org.apache.amoro.producer.spi.LakeFormatAdapter
+
+object LakehouseProducer {
+
+ private[producer] val adapters: Seq[LakeFormatAdapter] = Seq(PaimonFormatAdapter)
+
+ def main(args: Array[String]): Unit = {
+ ProducerApp.run(args, adapters)
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonCompactManifest.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonCompactManifest.scala
deleted file mode 100644
index 3d7e82a4b..000000000
--- a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonCompactManifest.scala
+++ /dev/null
@@ -1,424 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.amoro.producer
-
-import java.util.Locale
-
-import scala.util.control.NonFatal
-
-import org.apache.spark.sql.{Row, SparkSession}
-
-object PaimonCompactManifest {
-
- private val DefaultCatalogName = "paimon"
- private val DefaultRetryTimes = 3
- private val DefaultContinueOnTableFailure = true
-
- final private case class AppConfig(
- catalogName: String,
- databaseName: Option[String],
- tableName: Option[String],
- retryTimes: Int,
- continueOnTableFailure: Boolean,
- procedureOptions: String)
-
- final private case class TargetTable(raw: String, database: String, table: String)
-
- final private case class CompactResult(result: Boolean, costMs: Double)
-
- def main(args: Array[String]): Unit = {
- val config =
- try {
- val parsed = parseArgs(args)
- validateConfig(parsed)
- parsed
- } catch {
- case e: IllegalArgumentException =>
- printUsage()
- println(s"参数错误: ${e.getMessage}")
- sys.exit(2)
- }
-
- val spark = SparkSession.builder().getOrCreate()
- try {
- val targets = resolveTargetTables(spark, config)
- val modeDesc =
- if (targets.size > 1 || config.tableName.isEmpty) {
- "整库 Manifest 合并"
- } else {
- "单表 Manifest 合并"
- }
-
- if (targets.nonEmpty) {
- if (targets.size > 1 || config.tableName.isEmpty) {
- val databaseName = config.databaseName.getOrElse(targets.head.database)
- println(
- s"${modeDesc}启动,catalog=${config.catalogName}, 数据库 $databaseName 共发现 ${targets.size} 张表:")
- targets.foreach(t => println(s" - ${t.raw}"))
- } else {
- println(s"${modeDesc}启动,catalog=${config.catalogName}, 目标表: ${targets.head.raw}")
- }
- }
-
- var successTables = 0
- var failedTables = 0
- val allStartNs = System.nanoTime()
-
- targets.foreach { table =>
- try {
- compactOneTable(spark, config, table)
- successTables += 1
- } catch {
- case NonFatal(t) =>
- failedTables += 1
- println(s"[表 ${table.raw} Manifest 合并失败] ${t.getClass.getSimpleName}: ${t.getMessage}")
- if (!config.continueOnTableFailure) {
- throw t
- }
- }
- }
-
- val allCostMs = (System.nanoTime() - allStartNs) / 1e6
- println(
- f"[Manifest 合并汇总] catalog=${config.catalogName}, 模式=$modeDesc, " +
- f"successTables=$successTables, failedTables=$failedTables, " +
- f"总耗时=${allCostMs}%.3f ms, 表数=${targets.size}")
- } finally {
- spark.stop()
- }
- }
-
- private def resolveTargetTables(spark: SparkSession, config: AppConfig): Seq[TargetTable] = {
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- Seq(makeTarget(
- requireNonEmpty(database, "--databaseName"),
- requireNonEmpty(table, "--tableName")))
-
- case (Some(database), None) =>
- listTables(spark, config.catalogName, requireNonEmpty(database, "--databaseName"))
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
- Seq(makeTarget(parts(0).trim, parts(1).trim))
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
- }
-
- private def listTables(
- spark: SparkSession,
- catalogName: String,
- database: String): Seq[TargetTable] = {
- val showTablesSql = s"SHOW TABLES IN ${quoteIdent(catalogName)}.${quoteIdent(database)}"
- println(s"listing tables: $showTablesSql")
-
- val rows = spark.sql(showTablesSql).collect()
- rows.flatMap { row =>
- val tableNameOpt = fieldOpt(row, Set("tablename", "table"))
- .orElse(valueAt(row, 1))
- .filter(_ != null)
- .map(_.toString.trim)
-
- val isTemporary = fieldOpt(row, Set("istemporary", "temporary")).exists {
- case b: java.lang.Boolean => b.booleanValue()
- case s: String => s.equalsIgnoreCase("true") || s == "1"
- case i: java.lang.Integer => i == 1
- case l: java.lang.Long => l == 1L
- case _ => false
- }
-
- tableNameOpt
- .filter(name => name.nonEmpty && !isTemporary)
- .map(name => makeTarget(database, name))
- }.toSeq
- }
-
- private def compactOneTable(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): CompactResult = {
- println(s"========== 表 ${table.raw} 开始 Manifest 合并 ==========")
- val startNs = System.nanoTime()
- val result = runOnceWithRetry(spark, config, table)
- val costMs = (System.nanoTime() - startNs) / 1e6
-
- println(
- f"[表 ${table.raw} Manifest 合并结束] " +
- f"耗时=${costMs}%.3f ms, result=$result")
-
- CompactResult(result, costMs)
- }
-
- private def runOnceWithRetry(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): Boolean = {
- var attempt = 1
- var lastError: Throwable = null
-
- while (attempt <= config.retryTimes) {
- try {
- println(s"[表 ${table.raw}] 第 $attempt 次执行 compact_manifest")
- return runCompactManifest(spark, config, table)
- } catch {
- case NonFatal(t) =>
- lastError = t
- println(
- s"[表 ${table.raw}] 第 $attempt 次执行失败: ${t.getClass.getSimpleName}: ${t.getMessage}")
- attempt += 1
- }
- }
-
- throw lastError
- }
-
- private def runCompactManifest(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): Boolean = {
- val sqlText = buildCompactManifestSql(config, table)
- println(s"executing compact_manifest, table = ${table.raw}")
- println(sqlText)
-
- val resultRows = spark.sql(sqlText).collect()
- val result = resultRows.headOption
- .flatMap(row => fieldOpt(row, Set("result")).orElse(valueAt(row, 0)))
- .exists(toBoolean)
-
- println(s"SUCCESS table = ${table.raw}, result = $result")
- result
- }
-
- private def buildCompactManifestSql(config: AppConfig, table: TargetTable): String = {
- val optionsClause =
- if (config.procedureOptions.trim.isEmpty) {
- ""
- } else {
- s",\n options => ${sqlString(config.procedureOptions)}"
- }
-
- s"""
- |CALL ${quoteIdent(config.catalogName)}.sys.compact_manifest(
- | table => ${sqlString(table.raw)}$optionsClause
- |)
- |""".stripMargin
- }
-
- private def makeTarget(database: String, table: String): TargetTable = {
- TargetTable(s"${database}.${table}", database, table)
- }
-
- private def validateConfig(config: AppConfig): Unit = {
- requireNonEmpty(config.catalogName, "--catalogName")
-
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- requireNonEmpty(database, "--databaseName")
- val trimTable = requireNonEmpty(table, "--tableName")
- if (trimTable.contains(".")) {
- throw new IllegalArgumentException(
- "当 --databaseName 与 --tableName 同时提供时,--tableName 仅允许传入表名,不允许使用 db.table")
- }
-
- case (Some(database), None) =>
- requireNonEmpty(database, "--databaseName")
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
-
- if (config.retryTimes <= 0) {
- throw new IllegalArgumentException("--retryTimes 必须大于 0")
- }
- }
-
- private def parseArgs(args: Array[String]): AppConfig = {
- var catalogName = DefaultCatalogName
- var databaseName: Option[String] = None
- var tableName: Option[String] = None
- var retryTimes = DefaultRetryTimes
- var continueOnTableFailure = DefaultContinueOnTableFailure
- var procedureOptions = ""
-
- var i = 0
- while (i < args.length) {
- args(i) match {
- case "--help" | "-h" =>
- printUsage()
- sys.exit(0)
-
- case "--catalogName" =>
- catalogName = readValue(args, i, "--catalogName")
- i += 1
-
- case "--databaseName" =>
- databaseName = Some(readValue(args, i, "--databaseName"))
- i += 1
-
- case "--tableName" =>
- tableName = Some(readValue(args, i, "--tableName"))
- i += 1
-
- case "--retryTimes" =>
- retryTimes = parseInt(readValue(args, i, "--retryTimes"), "--retryTimes")
- i += 1
-
- case "--continueOnTableFailure" =>
- continueOnTableFailure =
- parseBoolean(readValue(args, i, "--continueOnTableFailure"), "--continueOnTableFailure")
- i += 1
-
- case "--procedureOptions" =>
- procedureOptions = readValue(args, i, "--procedureOptions")
- i += 1
-
- case other =>
- throw new IllegalArgumentException(s"未知参数: $other")
- }
- i += 1
- }
-
- AppConfig(
- catalogName,
- databaseName,
- tableName,
- retryTimes,
- continueOnTableFailure,
- procedureOptions)
- }
-
- private def readValue(args: Array[String], index: Int, argName: String): String = {
- if (index + 1 >= args.length || args(index + 1).startsWith("--")) {
- throw new IllegalArgumentException(s"$argName 缺少参数值")
- }
- args(index + 1)
- }
-
- private def parseInt(value: String, argName: String): Int = {
- try {
- value.toInt
- } catch {
- case _: NumberFormatException =>
- throw new IllegalArgumentException(s"$argName 必须是整数")
- }
- }
-
- private def parseBoolean(value: String, argName: String): Boolean = {
- value.trim.toLowerCase(Locale.ROOT) match {
- case "true" => true
- case "false" => false
- case _ => throw new IllegalArgumentException(s"$argName 必须是 true 或 false")
- }
- }
-
- private def requireNonEmpty(value: String, argName: String): String = {
- val trimmed = value.trim
- if (trimmed.isEmpty) {
- throw new IllegalArgumentException(s"$argName 的值不能为空")
- }
- trimmed
- }
-
- private def quoteIdent(name: String): String = {
- "`" + name.replace("`", "``") + "`"
- }
-
- private def sqlString(value: String): String = {
- "'" + value.replace("'", "''") + "'"
- }
-
- private def fieldOpt(row: Row, normalizedNames: Set[String]): Option[Any] = {
- val schemaOpt =
- try {
- Option(row.schema)
- } catch {
- case NonFatal(_) => None
- }
-
- schemaOpt.flatMap { schema =>
- schema.fields.zipWithIndex
- .find { case (field, _) => normalizedNames.contains(normalizeFieldName(field.name)) }
- .flatMap { case (_, index) => valueAt(row, index) }
- }
- }
-
- private def valueAt(row: Row, index: Int): Option[Any] = {
- if (index >= 0 && index < row.size && !row.isNullAt(index)) {
- Some(row.get(index))
- } else {
- None
- }
- }
-
- private def normalizeFieldName(name: String): String = {
- name.toLowerCase(Locale.ROOT).replace("_", "")
- }
-
- private def toBoolean(value: Any): Boolean = {
- value match {
- case null => false
- case b: java.lang.Boolean => b.booleanValue()
- case s: String => s.trim.equalsIgnoreCase("true") || s.trim == "1"
- case i: java.lang.Integer => i == 1
- case l: java.lang.Long => l == 1L
- case other => other.toString.trim.equalsIgnoreCase("true")
- }
- }
-
- private def printUsage(): Unit = {
- println(
- s"""
- |Usage:
- | spark-submit --class org.apache.amoro.producer.PaimonCompactManifest <jar> [options]
- |
- |Required target options:
- | --databaseName <db> 合并指定库下全部非临时表 Manifest
- | --databaseName <db> --tableName <t> 只合并指定库下的单表 Manifest
- | --tableName <db.table> 只合并全限定单表 Manifest
- |
- |Optional options:
- | --catalogName <catalog> Paimon catalog 名称,默认: $DefaultCatalogName
- | --procedureOptions <options> compact_manifest options,默认不传
- | --retryTimes <num> 单表失败最大执行次数,默认: $DefaultRetryTimes
- | --continueOnTableFailure <true|false>
- | 单表失败后是否继续后续表,默认: $DefaultContinueOnTableFailure
- | --help, -h 打印帮助信息
- |
- |Examples:
- | --databaseName sl_oki_test
- | --databaseName sl_oki_test --tableName t_order
- | --tableName sl_oki_test.t_order
- |""".stripMargin)
- }
-}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonExpireSnapshots.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonExpireSnapshots.scala
deleted file mode 100644
index 66f51245c..000000000
--- a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonExpireSnapshots.scala
+++ /dev/null
@@ -1,491 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.amoro.producer
-
-import java.util.Locale
-
-import scala.util.control.NonFatal
-
-import org.apache.spark.sql.{Row, SparkSession}
-
-object PaimonExpireSnapshots {
-
- private val DefaultRetainMax = 20
- private val DefaultRetainMin = 10
- private val DefaultMaxDeletes = 550L
- private val DefaultRetryTimes = 3
- private val DefaultContinueOnTableFailure = true
- private val DefaultProcedureOptions =
- "file-operation.thread-num=32,snapshot.expire.execution-mode=sync"
-
- final private case class AppConfig(
- databaseName: Option[String],
- tableName: Option[String],
- retainMax: Int,
- retainMin: Int,
- maxDeletes: Long,
- retryTimes: Int,
- continueOnTableFailure: Boolean,
- procedureOptions: String)
-
- final private case class TargetTable(raw: String, database: String, table: String)
-
- def main(args: Array[String]): Unit = {
- val config =
- try {
- val parsed = parseArgs(args)
- validateConfig(parsed)
- parsed
- } catch {
- case e: IllegalArgumentException =>
- printUsage()
- println(s"参数错误: ${e.getMessage}")
- sys.exit(2)
- }
-
- val spark = SparkSession.builder().getOrCreate()
- try {
- val targets = resolveTargetTables(spark, config)
-
- val (targetsByMode, modeDesc) =
- if (targets.size > 1 || config.tableName.isEmpty) {
- val databaseName = config.databaseName.getOrElse("unknown")
- (Some(databaseName), "整库清理")
- } else {
- (None, "单表清理")
- }
-
- if (targets.nonEmpty) {
- if (targetsByMode.nonEmpty) {
- println(s"${modeDesc}启动,数据库 ${targetsByMode.get} 共发现 ${targets.size} 张表:")
- targets.foreach(t => println(s" - ${t.raw}"))
- } else {
- println(s"${modeDesc}启动,目标表: ${targets.head.raw}")
- }
- }
-
- var successTables = 0
- var failedTables = 0
- val allStartNs = System.nanoTime()
-
- targets.foreach { table =>
- try {
- expireOneTable(spark, config, table)
- successTables += 1
- } catch {
- case NonFatal(t) =>
- failedTables += 1
- println(s"[表 ${table.raw} 清理失败] ${t.getClass.getSimpleName}: ${t.getMessage}")
- if (!config.continueOnTableFailure) {
- throw t
- }
- }
- }
-
- val allCostMs = (System.nanoTime() - allStartNs) / 1e6
- println(
- f"[清理汇总] 模式=$modeDesc, successTables=$successTables, failedTables=$failedTables, " +
- f"总耗时=${allCostMs}%.3f ms, 表数=${targets.size}")
- } finally {
- spark.stop()
- }
- }
-
- private def resolveTargetTables(spark: SparkSession, config: AppConfig): Seq[TargetTable] = {
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- Seq(makeTarget(
- requireNonEmpty(database, "--databaseName"),
- requireNonEmpty(table, "--tableName")))
-
- case (Some(database), None) =>
- listTables(spark, requireNonEmpty(database, "--databaseName"))
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
- if (parts.length == 2) {
- Seq(makeTarget(parts(0).trim, parts(1).trim))
- } else {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
- }
-
- private def requireNonEmpty(value: String, argName: String): String = {
- val trimmed = value.trim
- if (trimmed.isEmpty) {
- throw new IllegalArgumentException(s"$argName 的值不能为空")
- }
- trimmed
- }
-
- private def validateConfig(config: AppConfig): Unit = {
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- requireNonEmpty(database, "--databaseName")
- val trimTable = requireNonEmpty(table, "--tableName")
- if (trimTable.contains(".")) {
- throw new IllegalArgumentException(
- "当 --databaseName 与 --tableName 同时提供时,--tableName 仅允许传入表名,不允许使用 db.table")
- }
-
- case (Some(database), None) =>
- requireNonEmpty(database, "--databaseName")
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
-
- if (config.retainMax < 0) {
- throw new IllegalArgumentException("--retainMax 不能小于 0")
- }
- if (config.retainMin < 0) {
- throw new IllegalArgumentException("--retainMin 不能小于 0")
- }
- if (config.maxDeletes < 0) {
- throw new IllegalArgumentException("--maxDeletes 不能小于 0")
- }
- if (config.retryTimes <= 0) {
- throw new IllegalArgumentException("--retryTimes 必须大于 0")
- }
- }
-
- private def makeTarget(database: String, table: String): TargetTable = {
- TargetTable(s"${database}.${table}", database, table)
- }
-
- private def listTables(spark: SparkSession, database: String): Seq[TargetTable] = {
- val df = spark.sql(s"SHOW TABLES IN ${quoteIdent(database)}")
- val rows = df.collect()
- val tables = rows.flatMap { row =>
- val tableNameOpt = fieldOpt(row, Set("tablename", "table_name", "table"))
- .orElse(if (row.size > 1) {
- Option(row.get(1))
- } else {
- None
- })
- .filter(_ != null)
- .map(_.toString)
-
- val isTemporary = fieldOpt(row, Set("istemporary", "is_temporary")).exists {
- case b: java.lang.Boolean => b.booleanValue()
- case s: String => s.equalsIgnoreCase("true") || s == "1"
- case i: java.lang.Integer => i == 1
- case l: java.lang.Long => l == 1L
- case _ => false
- }
-
- tableNameOpt
- .filter(name => name != null && name.trim.nonEmpty && !isTemporary)
- .map(name => makeTarget(database, name))
- }
- tables.toSeq
- }
-
- private def expireOneTable(spark: SparkSession, config: AppConfig, table: TargetTable): Unit = {
- var round = 1
- var executedRounds = 0
- var totalCostMs = 0.0
- var maxCostMs = 0.0
- var minCostMs = Double.PositiveInfinity
- var shouldContinue = true
-
- println(s"========== 表 ${table.raw} 开始清理 ==========")
- while (shouldContinue) {
- println(s"========== 表 ${table.raw} 开始第 ${round} 轮过期快照清理 ==========")
- val roundStartNs = System.nanoTime()
-
- val deletedCount = runOnceWithRetry(spark, config, table.raw)
- val roundCostMs = (System.nanoTime() - roundStartNs) / 1e6
- executedRounds += 1
- totalCostMs += roundCostMs
- maxCostMs = math.max(maxCostMs, roundCostMs)
- minCostMs = math.min(minCostMs, roundCostMs)
-
- val avgCostMs = totalCostMs / executedRounds
- shouldContinue = deletedCount >= config.maxDeletes
-
- println(
- f"[表 ${table.raw} 第 ${round} 轮统计] " +
- f"本轮耗时=${roundCostMs}%.3f ms, " +
- f"平均耗时=${avgCostMs}%.3f ms, " +
- f"最大耗时=${maxCostMs}%.3f ms, " +
- f"最小耗时=${minCostMs}%.3f ms, " +
- f"deletedCount=$deletedCount, continue=$shouldContinue")
- round += 1
- }
-
- println(
- f"[表 ${table.raw} 清理结束] " +
- f"总轮次=$executedRounds, " +
- f"平均耗时=${if (executedRounds == 0) 0.0 else totalCostMs / executedRounds}%.3f ms, " +
- f"最大耗时=${if (executedRounds == 0) 0.0 else maxCostMs}%.3f ms, " +
- f"最小耗时=${if (executedRounds == 0) 0.0 else minCostMs}%.3f ms")
- }
-
- private def runOnceWithRetry(
- spark: SparkSession,
- config: AppConfig,
- tableName: String): Long = {
- var attempt = 1
- var lastError: Throwable = null
-
- while (attempt <= config.retryTimes) {
- try {
- val df = spark.sql(buildExpireSql(tableName, config))
- val rows = df.collect()
-
- if (rows.isEmpty) {
- throw new IllegalStateException(s"expire_snapshots 返回空结果,table=$tableName")
- }
-
- val deletedCount = parseDeletedCount(rows(0))
- println(
- s"[表 $tableName 当前轮尝试 ${attempt}/${config.retryTimes}] " +
- s"成功,schema=${df.schema.simpleString}, " +
- s"row=${rows(0).mkString("[", ", ", "]")}, deletedCount=$deletedCount")
- return deletedCount
- } catch {
- case NonFatal(t) =>
- lastError = t
- println(
- s"[表 $tableName 当前轮尝试 ${attempt}/${config.retryTimes}] 失败: " +
- s"${t.getClass.getSimpleName}: ${t.getMessage}")
- if (attempt == config.retryTimes) {
- throw new RuntimeException(
- s"表 $tableName expire_snapshots 连续失败 ${config.retryTimes} 次",
- lastError)
- }
- attempt += 1
- }
- }
- throw new RuntimeException("不应到达此分支")
- }
-
- private def buildExpireSql(tableName: String, config: AppConfig): String = {
- val escapedTableName = escapeSqlString(tableName)
- val escapedOptions = escapeSqlString(config.procedureOptions)
- val optionsSql =
- if (config.procedureOptions.trim.isEmpty) "" else s", options => '$escapedOptions'"
- s"""
- |CALL sys.expire_snapshots(
- | table => '$escapedTableName',
- | retain_max => ${config.retainMax},
- | retain_min => ${config.retainMin},
- | max_deletes => ${config.maxDeletes}
- |$optionsSql
- |)
- |""".stripMargin
- }
-
- private def quoteIdent(s: String): String = "`" + s.replace("`", "``") + "`"
-
- private def escapeSqlString(value: String): String = value.replace("'", "''")
-
- private def parseDeletedCount(row: Row): Long = {
- val preferredNames = Set(
- "deleted_count",
- "deletedcount",
- "deleted_snapshots",
- "deleted_snapshots_count",
- "deletedsnapshotcount",
- "result")
-
- val byName =
- try {
- val schema = row.schema
- schema.fieldNames.zipWithIndex.collectFirst {
- case (name, idx) if preferredNames.contains(name.toLowerCase(Locale.ROOT)) =>
- row.get(idx)
- }.flatMap(toLongOpt)
- } catch {
- case NonFatal(_) => None
- }
-
- byName
- .orElse(row.toSeq.flatMap(toLongOpt).headOption)
- .getOrElse {
- throw new IllegalStateException(
- s"无法从返回结果解析删除数量,row=${row.mkString("[", ", ", "]")}")
- }
- }
-
- private def toLongOpt(v: Any): Option[Long] = v match {
- case null => None
- case n: java.lang.Number => Some(n.longValue())
- case s: String if s.trim.matches("-?\\d+") => Some(s.trim.toLong)
- case _ => None
- }
-
- private def fieldOpt(row: Row, names: Set[String]): Option[Any] = {
- try {
- val schema = row.schema
- schema.fieldNames.zipWithIndex.collectFirst {
- case (name, idx) if names.contains(name.toLowerCase(Locale.ROOT)) => row.get(idx)
- }
- } catch {
- case NonFatal(_) => None
- }
- }
-
- private def parseArgs(args: Array[String]): AppConfig = {
- var databaseName: Option[String] = None
- var tableName: Option[String] = None
- var retainMax = DefaultRetainMax
- var retainMin = DefaultRetainMin
- var maxDeletes = DefaultMaxDeletes
- var retryTimes = DefaultRetryTimes
- var continueOnTableFailure = DefaultContinueOnTableFailure
- var procedureOptions = DefaultProcedureOptions
-
- var i = 0
- while (i < args.length) {
- args(i) match {
- case "--help" | "-h" =>
- printUsage()
- sys.exit(0)
- case "--databaseName" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException(s"--databaseName 需要值")
- }
- databaseName = Some(args(i))
- case "--tableName" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException(s"--tableName 需要值")
- }
- tableName = Some(args(i))
- case "--retainMax" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--retainMax 需要值")
- }
- retainMax = parseIntArg("--retainMax", args(i))
- case "--retainMin" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--retainMin 需要值")
- }
- retainMin = parseIntArg("--retainMin", args(i))
- case "--maxDeletes" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--maxDeletes 需要值")
- }
- maxDeletes = parseLongArg("--maxDeletes", args(i))
- case "--retryTimes" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--retryTimes 需要值")
- }
- retryTimes = parseIntArg("--retryTimes", args(i))
- case "--continueOnTableFailure" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--continueOnTableFailure 需要 true/false")
- }
- continueOnTableFailure = parseBooleanArg("--continueOnTableFailure", args(i))
- case "--procedureOptions" =>
- i += 1
- if (i >= args.length) {
- throw new IllegalArgumentException("--procedureOptions 需要值")
- }
- procedureOptions = args(i)
- case value if value.startsWith("--") =>
- throw new IllegalArgumentException(s"未知参数: $value")
- case value =>
- throw new IllegalArgumentException(s"参数格式错误: $value")
- }
- i += 1
- }
-
- AppConfig(
- databaseName,
- tableName,
- retainMax,
- retainMin,
- maxDeletes,
- retryTimes,
- continueOnTableFailure,
- procedureOptions)
- }
-
- private def parseIntArg(argName: String, value: String): Int = {
- try {
- value.toInt
- } catch {
- case NonFatal(_) => throw new IllegalArgumentException(s"$argName 必须是整数,当前: $value")
- }
- }
-
- private def parseLongArg(argName: String, value: String): Long = {
- try {
- value.toLong
- } catch {
- case NonFatal(_) => throw new IllegalArgumentException(s"$argName 必须是整数,当前: $value")
- }
- }
-
- private def parseBooleanArg(argName: String, value: String): Boolean = {
- value.toLowerCase(Locale.ROOT) match {
- case "true" | "1" | "yes" => true
- case "false" | "0" | "no" => false
- case _ =>
- throw new IllegalArgumentException(
- s"$argName 仅支持 true/false/1/0/yes/no,当前: $value")
- }
- }
-
- private def printUsage(): Unit = {
- println(
- """
- |Usage:
- | spark-submit --class org.apache.amoro.producer.PaimonExpireSnapshots <jar> [options]
- |
- |Options:
- | --databaseName <database> 指定数据库名(与 --tableName 不同时提供则扫描全库)
- | --tableName <table> 指定表名(与 --databaseName 同时使用时只清理该表;仅单独使用时需 db.table)
- | --retainMax <int> 保留快照数上限,默认 20
- | --retainMin <int> 保留快照下限,默认 10
- | --maxDeletes <long> 单次最多删除快照数,默认 550
- | --retryTimes <int> SQL 重试次数,默认 3
- | --continueOnTableFailure <bool> 出错是否继续,默认 true
- | --procedureOptions <string> expire_snapshots options 字符串,默认 file-operation.thread-num=32,snapshot.expire.execution-mode=sync
- | --help, -h 打印帮助
- |""".stripMargin)
- }
-}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonRemoveOrphanFiles.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonRemoveOrphanFiles.scala
deleted file mode 100644
index 4a754876f..000000000
--- a/amoro-producer/src/main/scala/org/apache/amoro/producer/PaimonRemoveOrphanFiles.scala
+++ /dev/null
@@ -1,478 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.amoro.producer
-
-import java.util.Locale
-
-import scala.util.control.NonFatal
-
-import org.apache.spark.sql.{Row, SparkSession}
-
-object PaimonRemoveOrphanFiles {
-
- private val DefaultCatalogName = "paimon"
- private val DefaultParallelism = 10
- private val DefaultMode = "distributed"
- private val DefaultRetryTimes = 3
- private val DefaultContinueOnTableFailure = true
-
- final private case class AppConfig(
- catalogName: String,
- databaseName: Option[String],
- tableName: Option[String],
- parallelism: Int,
- mode: String,
- retryTimes: Int,
- continueOnTableFailure: Boolean)
-
- final private case class TargetTable(raw: String, database: String, table: String)
-
- final private case class CleanResult(
- deletedFileCount: Long,
- deletedFileTotalLenInBytes: Long,
- costMs: Double)
-
- def main(args: Array[String]): Unit = {
- val config =
- try {
- val parsed = parseArgs(args)
- validateConfig(parsed)
- parsed
- } catch {
- case e: IllegalArgumentException =>
- printUsage()
- println(s"参数错误: ${e.getMessage}")
- sys.exit(2)
- }
-
- val spark = SparkSession.builder().getOrCreate()
- try {
- val targets = resolveTargetTables(spark, config)
- val modeDesc =
- if (targets.size > 1 || config.tableName.isEmpty) {
- "整库孤儿文件清理"
- } else {
- "单表孤儿文件清理"
- }
-
- if (targets.nonEmpty) {
- if (targets.size > 1 || config.tableName.isEmpty) {
- val databaseName = config.databaseName.getOrElse(targets.head.database)
- println(
- s"${modeDesc}启动,catalog=${config.catalogName}, 数据库 $databaseName 共发现 ${targets.size} 张表:")
- targets.foreach(t => println(s" - ${t.raw}"))
- } else {
- println(s"${modeDesc}启动,catalog=${config.catalogName}, 目标表: ${targets.head.raw}")
- }
- }
-
- var successTables = 0
- var failedTables = 0
- var totalDeletedFileCount = 0L
- var totalDeletedFileTotalLenInBytes = 0L
- val allStartNs = System.nanoTime()
-
- targets.foreach { table =>
- try {
- val result = removeOneTable(spark, config, table)
- successTables += 1
- totalDeletedFileCount += result.deletedFileCount
- totalDeletedFileTotalLenInBytes += result.deletedFileTotalLenInBytes
- } catch {
- case NonFatal(t) =>
- failedTables += 1
- println(s"[表 ${table.raw} 孤儿文件清理失败] ${t.getClass.getSimpleName}: ${t.getMessage}")
- if (!config.continueOnTableFailure) {
- throw t
- }
- }
- }
-
- val allCostMs = (System.nanoTime() - allStartNs) / 1e6
- println("=========== ORPHAN FILES CLEAN SUMMARY ===========")
- println(s"catalog = ${config.catalogName}")
- println(s"mode = $modeDesc")
- println(s"tables scanned = ${targets.size}")
- println(s"tables success = $successTables")
- println(s"tables failed = $failedTables")
- println(s"TOTAL deleted files = $totalDeletedFileCount")
- println(
- s"TOTAL deleted bytes = $totalDeletedFileTotalLenInBytes (${humanBytes(totalDeletedFileTotalLenInBytes)})")
- println(f"total cost = ${allCostMs}%.3f ms")
- println("==================================================")
- } finally {
- spark.stop()
- }
- }
-
- private def resolveTargetTables(spark: SparkSession, config: AppConfig): Seq[TargetTable] = {
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- Seq(makeTarget(
- requireNonEmpty(database, "--databaseName"),
- requireNonEmpty(table, "--tableName")))
-
- case (Some(database), None) =>
- listTables(spark, config.catalogName, requireNonEmpty(database, "--databaseName"))
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
- Seq(makeTarget(parts(0).trim, parts(1).trim))
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
- }
-
- private def listTables(
- spark: SparkSession,
- catalogName: String,
- database: String): Seq[TargetTable] = {
- val showTablesSql = s"SHOW TABLES IN ${quoteIdent(catalogName)}.${quoteIdent(database)}"
- println(s"listing tables: $showTablesSql")
-
- val rows = spark.sql(showTablesSql).collect()
- rows.flatMap { row =>
- val tableNameOpt = fieldOpt(row, Set("tablename", "table"))
- .orElse(valueAt(row, 1))
- .filter(_ != null)
- .map(_.toString.trim)
-
- val isTemporary = fieldOpt(row, Set("istemporary", "temporary")).exists {
- case b: java.lang.Boolean => b.booleanValue()
- case s: String => s.equalsIgnoreCase("true") || s == "1"
- case i: java.lang.Integer => i == 1
- case l: java.lang.Long => l == 1L
- case _ => false
- }
-
- tableNameOpt
- .filter(name => name.nonEmpty && !isTemporary)
- .map(name => makeTarget(database, name))
- }.toSeq
- }
-
- private def removeOneTable(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): CleanResult = {
- println(s"========== 表 ${table.raw} 开始孤儿文件清理 ==========")
- val startNs = System.nanoTime()
- val (deletedFileCount, deletedFileTotalLenInBytes) = runOnceWithRetry(spark, config, table)
- val costMs = (System.nanoTime() - startNs) / 1e6
-
- println(
- f"[表 ${table.raw} 孤儿文件清理结束] " +
- f"耗时=${costMs}%.3f ms, " +
- f"deletedFileCount=$deletedFileCount, " +
- f"deletedFileTotalLenInBytes=$deletedFileTotalLenInBytes, " +
- f"humanDeletedBytes=${humanBytes(deletedFileTotalLenInBytes)}")
-
- CleanResult(deletedFileCount, deletedFileTotalLenInBytes, costMs)
- }
-
- private def runOnceWithRetry(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): (Long, Long) = {
- var attempt = 1
- var lastError: Throwable = null
-
- while (attempt <= config.retryTimes) {
- try {
- println(s"[表 ${table.raw}] 第 $attempt 次执行 remove_orphan_files")
- return runRemoveOrphanFiles(spark, config, table)
- } catch {
- case NonFatal(t) =>
- lastError = t
- println(
- s"[表 ${table.raw}] 第 $attempt 次执行失败: ${t.getClass.getSimpleName}: ${t.getMessage}")
- attempt += 1
- }
- }
-
- throw lastError
- }
-
- private def runRemoveOrphanFiles(
- spark: SparkSession,
- config: AppConfig,
- table: TargetTable): (Long, Long) = {
- val sqlText = buildRemoveOrphanFilesSql(config, table)
- println(s"executing remove_orphan_files, table = ${table.raw}")
- println(sqlText)
-
- val resultRows = spark.sql(sqlText).collect()
- var deletedFileCount = 0L
- var deletedFileTotalLenInBytes = 0L
-
- resultRows.foreach { row =>
- val countValue = fieldOpt(row, Set("deletedfilecount"))
- .orElse(valueAt(row, 0))
- .getOrElse(0L)
- val bytesValue = fieldOpt(row, Set("deletedfiletotalleninbytes"))
- .orElse(valueAt(row, 1))
- .getOrElse(0L)
-
- deletedFileCount += toLong(countValue)
- deletedFileTotalLenInBytes += toLong(bytesValue)
- }
-
- println(
- s"SUCCESS table = ${table.raw}, deletedFileCount = $deletedFileCount, " +
- s"deletedFileTotalLenInBytes = $deletedFileTotalLenInBytes")
- (deletedFileCount, deletedFileTotalLenInBytes)
- }
-
- private def buildRemoveOrphanFilesSql(config: AppConfig, table: TargetTable): String = {
- s"""
- |CALL ${quoteIdent(config.catalogName)}.sys.remove_orphan_files(
- | table => ${sqlString(table.raw)},
- | parallelism => ${config.parallelism},
- | mode => ${sqlString(config.mode)}
- |)
- |""".stripMargin
- }
-
- private def makeTarget(database: String, table: String): TargetTable = {
- TargetTable(s"${database}.${table}", database, table)
- }
-
- private def validateConfig(config: AppConfig): Unit = {
- requireNonEmpty(config.catalogName, "--catalogName")
-
- (config.databaseName, config.tableName) match {
- case (Some(database), Some(table)) =>
- requireNonEmpty(database, "--databaseName")
- val trimTable = requireNonEmpty(table, "--tableName")
- if (trimTable.contains(".")) {
- throw new IllegalArgumentException(
- "当 --databaseName 与 --tableName 同时提供时,--tableName 仅允许传入表名,不允许使用 db.table")
- }
-
- case (Some(database), None) =>
- requireNonEmpty(database, "--databaseName")
-
- case (None, Some(table)) =>
- val trimTable = requireNonEmpty(table, "--tableName")
- val parts = trimTable.split("\\.", -1)
- if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
- throw new IllegalArgumentException(
- "只传入 --tableName 时,必须使用 db.table 格式,例如 --tableName db.table")
- }
-
- case (None, None) =>
- throw new IllegalArgumentException("参数不足:请至少提供 --databaseName 或 --tableName")
- }
-
- if (config.parallelism <= 0) {
- throw new IllegalArgumentException("--parallelism 必须大于 0")
- }
- requireNonEmpty(config.mode, "--mode")
- if (config.retryTimes <= 0) {
- throw new IllegalArgumentException("--retryTimes 必须大于 0")
- }
- }
-
- private def parseArgs(args: Array[String]): AppConfig = {
- var catalogName = DefaultCatalogName
- var databaseName: Option[String] = None
- var tableName: Option[String] = None
- var parallelism = DefaultParallelism
- var mode = DefaultMode
- var retryTimes = DefaultRetryTimes
- var continueOnTableFailure = DefaultContinueOnTableFailure
-
- var i = 0
- while (i < args.length) {
- args(i) match {
- case "--help" | "-h" =>
- printUsage()
- sys.exit(0)
-
- case "--catalogName" =>
- catalogName = readValue(args, i, "--catalogName")
- i += 1
-
- case "--databaseName" =>
- databaseName = Some(readValue(args, i, "--databaseName"))
- i += 1
-
- case "--tableName" =>
- tableName = Some(readValue(args, i, "--tableName"))
- i += 1
-
- case "--parallelism" =>
- parallelism = parseInt(readValue(args, i, "--parallelism"), "--parallelism")
- i += 1
-
- case "--mode" =>
- mode = readValue(args, i, "--mode")
- i += 1
-
- case "--retryTimes" =>
- retryTimes = parseInt(readValue(args, i, "--retryTimes"), "--retryTimes")
- i += 1
-
- case "--continueOnTableFailure" =>
- continueOnTableFailure =
- parseBoolean(readValue(args, i, "--continueOnTableFailure"), "--continueOnTableFailure")
- i += 1
-
- case other =>
- throw new IllegalArgumentException(s"未知参数: $other")
- }
- i += 1
- }
-
- AppConfig(
- catalogName,
- databaseName,
- tableName,
- parallelism,
- mode,
- retryTimes,
- continueOnTableFailure)
- }
-
- private def readValue(args: Array[String], index: Int, argName: String): String = {
- if (index + 1 >= args.length || args(index + 1).startsWith("--")) {
- throw new IllegalArgumentException(s"$argName 缺少参数值")
- }
- args(index + 1)
- }
-
- private def parseInt(value: String, argName: String): Int = {
- try {
- value.toInt
- } catch {
- case _: NumberFormatException =>
- throw new IllegalArgumentException(s"$argName 必须是整数")
- }
- }
-
- private def parseBoolean(value: String, argName: String): Boolean = {
- value.trim.toLowerCase(Locale.ROOT) match {
- case "true" => true
- case "false" => false
- case _ => throw new IllegalArgumentException(s"$argName 必须是 true 或 false")
- }
- }
-
- private def requireNonEmpty(value: String, argName: String): String = {
- val trimmed = value.trim
- if (trimmed.isEmpty) {
- throw new IllegalArgumentException(s"$argName 的值不能为空")
- }
- trimmed
- }
-
- private def quoteIdent(name: String): String = {
- "`" + name.replace("`", "``") + "`"
- }
-
- private def sqlString(value: String): String = {
- "'" + value.replace("'", "''") + "'"
- }
-
- private def fieldOpt(row: Row, normalizedNames: Set[String]): Option[Any] = {
- val schemaOpt =
- try {
- Option(row.schema)
- } catch {
- case NonFatal(_) => None
- }
-
- schemaOpt.flatMap { schema =>
- schema.fields.zipWithIndex
- .find { case (field, _) => normalizedNames.contains(normalizeFieldName(field.name)) }
- .flatMap { case (_, index) => valueAt(row, index) }
- }
- }
-
- private def valueAt(row: Row, index: Int): Option[Any] = {
- if (index >= 0 && index < row.size && !row.isNullAt(index)) {
- Some(row.get(index))
- } else {
- None
- }
- }
-
- private def normalizeFieldName(name: String): String = {
- name.toLowerCase(Locale.ROOT).replace("_", "")
- }
-
- private def toLong(value: Any): Long = {
- value match {
- case null => 0L
- case n: java.lang.Number => n.longValue()
- case s: String =>
- val trimmed = s.trim
- if (trimmed.isEmpty) {
- 0L
- } else {
- trimmed.toLong
- }
- case other => other.toString.toLong
- }
- }
-
- private def humanBytes(bytes: Long): String = {
- val units = Array("B", "KB", "MB", "GB", "TB", "PB")
- var value = bytes.toDouble
- var index = 0
- while (value >= 1024.0 && index < units.length - 1) {
- value = value / 1024.0
- index += 1
- }
- f"$value%.2f ${units(index)}"
- }
-
- private def printUsage(): Unit = {
- println(
- s"""
- |Usage:
- | spark-submit --class org.apache.amoro.producer.PaimonRemoveOrphanFiles <jar> [options]
- |
- |Required target options:
- | --databaseName <db> 清理指定库下全部非临时表
- | --databaseName <db> --tableName <t> 只清理指定库下的单表
- | --tableName <db.table> 只清理全限定单表
- |
- |Optional options:
- | --catalogName <catalog> Paimon catalog 名称,默认: $DefaultCatalogName
- | --parallelism <num> remove_orphan_files parallelism,默认: $DefaultParallelism
- | --mode <mode> remove_orphan_files mode,默认: $DefaultMode
- | --retryTimes <num> 单表失败最大执行次数,默认: $DefaultRetryTimes
- | --continueOnTableFailure <true|false>
- | 单表失败后是否继续后续表,默认: $DefaultContinueOnTableFailure
- | --help, -h 打印帮助信息
- |
- |Examples:
- | --databaseName sl_oki_test
- | --databaseName sl_oki_test --tableName t_order
- | --tableName sl_oki_test.t_order
- |""".stripMargin)
- }
-}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ActionResult.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ActionResult.scala
new file mode 100644
index 000000000..a5b0982de
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ActionResult.scala
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+sealed trait ActionStatus
+
+object ActionStatus {
+ case object Success extends ActionStatus
+ case object Failed extends ActionStatus
+ case object Skipped extends ActionStatus
+}
+
+final case class ActionTaskResult(
+ name: String,
+ status: ActionStatus,
+ message: Option[String],
+ metrics: Map[String, String])
+
+final case class ActionResult(
+ status: ActionStatus,
+ tasks: Seq[ActionTaskResult],
+ message: Option[String],
+ metrics: Map[String, String],
+ costMs: Double)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerApp.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerApp.scala
new file mode 100644
index 000000000..5dc6ada30
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerApp.scala
@@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.util.Locale
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+object ProducerApp {
+
+ def run(args: Array[String], adapters: Seq[LakeFormatAdapter]): Unit = {
+ val code = runInternal(args, adapters)
+ if (code != 0) {
+ sys.exit(code)
+ }
+ }
+
+ def runInternal(
+ args: Array[String],
+ adapters: Seq[LakeFormatAdapter],
+ sparkBuilder: () => SparkSession = () => SparkSession.builder().getOrCreate()): Int = {
+ val peek = ProducerCliParser.peek(args)
+ if (peek.help) {
+ printUsage(adapters, peek)
+ return 0
+ }
+
+ val (adapter, action, actionConfig, common) =
+ try {
+ val format =
+ peek.format.getOrElse {
+ throw new IllegalArgumentException("--format must be specified")
+ }
+ val actionName =
+ peek.action.getOrElse {
+ throw new IllegalArgumentException("--action must be specified")
+ }
+ val resolvedAdapter = resolveAdapter(format, adapters)
+ val resolvedAction = resolveAction(resolvedAdapter, actionName)
+ val commandLine = ProducerCliParser.parseMerged(resolvedAction.options, args)
+ val parsedCommon = ProducerCliParser.parseCommon(commandLine)
+ val parsedActionConfig = resolvedAction.parse(commandLine)
+ resolvedAction.validate(parsedCommon, parsedActionConfig)
+ TableResolver.validateTargetOptions(parsedCommon)
+ (resolvedAdapter, resolvedAction, parsedActionConfig, parsedCommon)
+ } catch {
+ case exception: ProducerCliParseException =>
+ printUsage(adapters, peek)
+ println(s"参数错误: ${exception.getMessage}")
+ return 2
+ case exception: IllegalArgumentException =>
+ printUsage(adapters, peek)
+ println(s"参数错误: ${exception.getMessage}")
+ return 2
+ }
+
+ var spark: SparkSession = null
+ try {
+ spark = sparkBuilder()
+ val targets = TableResolver.resolveTargets(spark, adapter, common)
+ printTargets(common, targets)
+ val context = ProducerContext(spark, common, adapter)
+ val results = executeTables(context, action, actionConfig, targets)
+ val summary = printSummary(common, results)
+ if (summary.failedTables.nonEmpty && !common.continueOnTableFailure) 1 else 0
+ } catch {
+ case NonFatal(exception) =>
+ println(s"执行失败: ${exception.getClass.getSimpleName}: ${exception.getMessage}")
+ 1
+ } finally {
+ if (spark != null) {
+ spark.stop()
+ }
+ }
+ }
+
+ def resolveAdapter(format: String, adapters: Seq[LakeFormatAdapter]): LakeFormatAdapter = {
+ val normalizedFormat = normalize(format)
+ requireUniqueNormalizedKeys(
+ values = adapters.map(_.format),
+ duplicatedMessage =
+ duplicate => s"Duplicate lakehouse format after normalization: $duplicate")
+ adapters
+ .find(adapter => normalize(adapter.format) == normalizedFormat)
+ .getOrElse {
+ throw new IllegalArgumentException(
+ s"Unsupported lakehouse format: $format. Available formats: ${adapters.map(_.format).mkString(", ")}")
+ }
+ }
+
+ def resolveAction(adapter: LakeFormatAdapter, actionName: String): ProducerAction = {
+ val normalizedAction = normalize(actionName)
+ val actions = adapter.actions
+ val availableActions = actions.map(_.name).mkString(", ")
+ requireUniqueNormalizedKeys(
+ values = actions.map(_.name),
+ duplicatedMessage = duplicate =>
+ s"Duplicate action name after normalization for format '${adapter.format}': $duplicate")
+ actions
+ .find(action => normalize(action.name) == normalizedAction)
+ .getOrElse {
+ throw new IllegalArgumentException(
+ s"Unsupported action '$actionName' for format '${adapter.format}'. Available actions: $availableActions")
+ }
+ }
+
+ def executeOneTableWithRetry(
+ context: ProducerContext,
+ action: ProducerAction,
+ actionConfig: ActionConfig,
+ table: TargetTable): ActionResult = {
+ action.retryMode match {
+ case RetryMode.ActionManagedRetry =>
+ action.execute(context, table, actionConfig)
+
+ case RetryMode.FrameworkTableRetry =>
+ executeWithFrameworkRetry(context, action, actionConfig, table)
+ }
+ }
+
+ def executeTables(
+ context: ProducerContext,
+ action: ProducerAction,
+ actionConfig: ActionConfig,
+ targets: Seq[TargetTable]): Seq[(TargetTable, ActionResult)] = {
+ validateRetryTimesForFrameworkRetry(context, action)
+ targets.map { table =>
+ val startNs = System.nanoTime()
+ val execution =
+ try {
+ Right(executeOneTableWithRetry(context, action, actionConfig, table))
+ } catch {
+ case NonFatal(exception) => Left(exception)
+ }
+
+ execution match {
+ case Right(actionResult) =>
+ val result = actionResult.copy(costMs = elapsedMs(startNs))
+ printTableAction(context.common, table, result)
+ if (result.status == ActionStatus.Failed && !context.common.continueOnTableFailure) {
+ throw new RuntimeException(failedStatusMessage(action, table, result))
+ }
+ table -> result
+
+ case Left(exception) =>
+ val result = failedResult(table, exception).copy(costMs = elapsedMs(startNs))
+ printTableAction(context.common, table, result)
+ if (context.common.continueOnTableFailure) {
+ table -> result
+ } else {
+ throw exception
+ }
+ }
+ }
+ }
+
+ private def executeWithFrameworkRetry(
+ context: ProducerContext,
+ action: ProducerAction,
+ actionConfig: ActionConfig,
+ table: TargetTable): ActionResult = {
+ validateRetryTimesForFrameworkRetry(context, action)
+ var attempt = 1
+ var lastFailure: Throwable = null
+ while (attempt <= context.common.retryTimes) {
+ try {
+ return action.execute(context, table, actionConfig)
+ } catch {
+ case NonFatal(exception) =>
+ lastFailure = exception
+ if (attempt == context.common.retryTimes) {
+ throw exception
+ }
+ }
+ attempt += 1
+ }
+ throw lastFailure
+ }
+
+ private def failedResult(table: TargetTable, exception: Throwable): ActionResult = {
+ ActionResult(
+ status = ActionStatus.Failed,
+ tasks = Seq.empty,
+ message = Some(Option(exception.getMessage).getOrElse(exception.getClass.getName)),
+ metrics = Map("table" -> table.raw),
+ costMs = 0.0d)
+ }
+
+ private def failedStatusMessage(
+ action: ProducerAction,
+ table: TargetTable,
+ result: ActionResult): String = {
+ val message = result.message.getOrElse("action returned failed status")
+ s"Action '${action.name}' failed for table ${table.raw}: $message"
+ }
+
+ private def printTableAction(
+ common: CommonProducerConfig,
+ table: TargetTable,
+ result: ActionResult): Unit = {
+ println(TableActionJsonLog.render(common, table, result, result.costMs))
+ }
+
+ private def elapsedMs(startNs: Long): Double = {
+ (System.nanoTime() - startNs) / 1e6
+ }
+
+ private def validateRetryTimesForFrameworkRetry(
+ context: ProducerContext,
+ action: ProducerAction): Unit = {
+ if (action.retryMode == RetryMode.FrameworkTableRetry && context.common.retryTimes <= 0) {
+ throw new IllegalArgumentException("retryTimes must be greater than 0")
+ }
+ }
+
+ private def requireUniqueNormalizedKeys(
+ values: Seq[String],
+ duplicatedMessage: String => String): Unit = {
+ val duplicateKeys =
+ values
+ .map(normalize)
+ .groupBy(identity)
+ .collect {
+ case (key, groupedValues) if groupedValues.size > 1 => key
+ }
+ .toSeq
+ .sorted
+
+ if (duplicateKeys.nonEmpty) {
+ throw new IllegalArgumentException(duplicatedMessage(duplicateKeys.mkString(", ")))
+ }
+ }
+
+ private def normalize(value: String): String = {
+ Option(value).map(_.trim.toLowerCase(Locale.ROOT)).getOrElse("")
+ }
+
+ private def printUsage(adapters: Seq[LakeFormatAdapter], peek: CliPeek): Unit = {
+ val options =
+ peek.format
+ .flatMap(format => adapters.find(adapter => normalize(adapter.format) == normalize(format)))
+ .flatMap { adapter =>
+ peek.action.flatMap(actionName =>
+ adapter.actions.find(action => normalize(action.name) == normalize(actionName)))
+ }
+ .map(_.options)
+ .getOrElse(Seq.empty)
+ println(ProducerCliParser.renderUsage(options, adapters.map(_.format)))
+ }
+
+ private def printTargets(common: CommonProducerConfig, targets: Seq[TargetTable]): Unit = {
+ if (targets.size > 1 || common.tableName.isEmpty) {
+ println(
+ s"Lakehouse producer started, format=${common.format}, action=${common.action}, " +
+ s"catalog=${common.catalogName}, tables=${targets.size}")
+ targets.foreach(table => println(s" - ${table.raw}"))
+ } else if (targets.nonEmpty) {
+ println(
+ s"Lakehouse producer started, format=${common.format}, action=${common.action}, " +
+ s"catalog=${common.catalogName}, table=${targets.head.raw}")
+ }
+ }
+
+ private def printSummary(
+ common: CommonProducerConfig,
+ results: Seq[(TargetTable, ActionResult)]): ProducerSummary = {
+ val summary = ProducerSummary.fromResults(results)
+ println("=========== PRODUCER SUMMARY ===========")
+ println(s"format = ${common.format}")
+ println(s"action = ${common.action}")
+ println(s"catalog = ${common.catalogName}")
+ println(s"tables found = ${summary.tableCount}")
+ println(s"tables success = ${summary.successTables.size}")
+ println(s"tables failed = ${summary.failedTables.size}")
+ println(s"tables skipped = ${summary.skippedTables.size}")
+ println(s"task success count = ${summary.taskSuccessCount}")
+ println(s"task failed count = ${summary.taskFailedCount}")
+ if (summary.failedTableSummaries.nonEmpty) {
+ println("failed tables:")
+ summary.failedTableSummaries.foreach { failedTable =>
+ println(s" - ${failedTable.table.raw} | ${failedTable.message}")
+ }
+ }
+ println("========================================")
+ summary
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerCliParser.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerCliParser.scala
new file mode 100644
index 000000000..e570e4086
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerCliParser.scala
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.util.Locale
+
+final case class CliPeek(format: Option[String], action: Option[String], help: Boolean)
+
+object ProducerCliParser {
+
+ private val DefaultCatalogName = "spark_catalog"
+ private val DefaultRetryTimes = 3
+ private val DefaultContinueOnTableFailure = true
+
+ def peek(args: Array[String]): CliPeek = {
+ var format: Option[String] = None
+ var action: Option[String] = None
+ var help = false
+ var index = 0
+
+ while (index < args.length) {
+ args(index) match {
+ case longOption if longOption.startsWith("--format=") =>
+ format = Some(longOption.substring("--format=".length))
+ case "--format" if index + 1 < args.length =>
+ format = Some(args(index + 1))
+ index += 1
+ case longOption if longOption.startsWith("--action=") =>
+ action = Some(longOption.substring("--action=".length))
+ case "--action" if index + 1 < args.length =>
+ action = Some(args(index + 1))
+ index += 1
+ case "--help" | "-h" =>
+ help = true
+ case _ =>
+ }
+ index += 1
+ }
+
+ CliPeek(format.map(_.trim).filter(_.nonEmpty), action.map(_.trim).filter(_.nonEmpty), help)
+ }
+
+ def commonOptions: Seq[ActionOption] = {
+ Seq(
+ ActionOption("format", description = "Lakehouse table format"),
+ ActionOption("action", description = "Producer action name"),
+ ActionOption("catalogName", description = "Spark catalog name"),
+ ActionOption("databaseName", description = "Target database name"),
+ ActionOption("tableName", description = "Target table name"),
+ ActionOption("tableNameRegex", description = "Target table name regex"),
+ ActionOption("retryTimes", description = "Retry times"),
+ ActionOption("continueOnTableFailure", description = "Continue when one table fails"),
+ ActionOption("help", hasValue = false, description = "Print help", shortName = Some("h")))
+ }
+
+ def mergeOptions(actionOptions: Seq[ActionOption]): Seq[ActionOption] = {
+ val common = commonOptions
+ val commonLongOptions = common.map(_.name).toSet
+ val commonShortOptions = common.flatMap(_.shortName).toSet
+ actionOptions.foreach { option =>
+ Option(option.name)
+ .filter(commonLongOptions.contains)
+ .foreach { longOpt =>
+ throw new IllegalArgumentException(s"Duplicate CLI long option: --$longOpt")
+ }
+ option.shortName
+ .filter(commonShortOptions.contains)
+ .foreach { shortOpt =>
+ throw new IllegalArgumentException(s"Duplicate CLI short option: -$shortOpt")
+ }
+ }
+ common ++ actionOptions
+ }
+
+ def parseMerged(actionOptions: Seq[ActionOption], args: Array[String]): ProducerParsedOptions = {
+ val options = mergeOptions(actionOptions)
+ val byLongName = options.map(option => option.name -> option).toMap
+ val byShortName = options.flatMap(option => option.shortName.map(_ -> option)).toMap
+ def isRegisteredOptionToken(token: String): Boolean = {
+ if (token.startsWith("--")) {
+ byLongName.contains(token.stripPrefix("--"))
+ } else if (token.startsWith("-") && token.length > 1) {
+ byShortName.contains(token.stripPrefix("-"))
+ } else {
+ false
+ }
+ }
+ var values = Map.empty[String, String]
+ var flags = Set.empty[String]
+ var index = 0
+
+ while (index < args.length) {
+ val token = args(index)
+ val longOptionWithValue =
+ if (token.startsWith("--")) {
+ val equalsIndex = token.indexOf('=')
+ if (equalsIndex > 2) {
+ Some(token.substring(2, equalsIndex) -> token.substring(equalsIndex + 1))
+ } else {
+ None
+ }
+ } else {
+ None
+ }
+ val option =
+ if (longOptionWithValue.nonEmpty) {
+ byLongName.getOrElse(
+ longOptionWithValue.get._1,
+ throw new ProducerCliParseException(
+ s"Unrecognized option: --${longOptionWithValue.get._1}"))
+ } else if (token.startsWith("--")) {
+ byLongName.getOrElse(
+ token.stripPrefix("--"),
+ throw new ProducerCliParseException(s"Unrecognized option: $token"))
+ } else if (token.startsWith("-") && token.length > 1) {
+ byShortName.getOrElse(
+ token.stripPrefix("-"),
+ throw new ProducerCliParseException(s"Unrecognized option: $token"))
+ } else {
+ throw new ProducerCliParseException(s"Unexpected argument: $token")
+ }
+
+ if (option.hasValue) {
+ val value =
+ longOptionWithValue.map(_._2).getOrElse {
+ if (index + 1 >= args.length) {
+ throw new ProducerCliParseException(s"Missing argument for option: --${option.name}")
+ }
+ val nextToken = args(index + 1)
+ if (isRegisteredOptionToken(nextToken)) {
+ throw new ProducerCliParseException(s"Missing argument for option: --${option.name}")
+ }
+ index += 1
+ nextToken
+ }
+ values += option.name -> value
+ } else {
+ if (longOptionWithValue.nonEmpty) {
+ throw new ProducerCliParseException(s"Option does not accept argument: --${option.name}")
+ }
+ flags += option.name
+ }
+ index += 1
+ }
+
+ ProducerParsedOptions(values, flags)
+ }
+
+ def parseCommon(commandLine: ProducerParsedOptions): CommonProducerConfig = {
+ val config =
+ CommonProducerConfig(
+ format = stringValue(commandLine.getOptionValue("format")),
+ action = stringValue(commandLine.getOptionValue("action")),
+ catalogName = stringValue(commandLine.getOptionValue("catalogName", DefaultCatalogName)),
+ databaseName = optionValue(commandLine, "databaseName"),
+ tableName = optionValue(commandLine, "tableName"),
+ tableNameRegex = optionValue(commandLine, "tableNameRegex"),
+ retryTimes =
+ parseInt(
+ commandLine.getOptionValue("retryTimes", DefaultRetryTimes.toString),
+ "--retryTimes"),
+ continueOnTableFailure =
+ parseBoolean(
+ commandLine.getOptionValue(
+ "continueOnTableFailure",
+ DefaultContinueOnTableFailure.toString),
+ "--continueOnTableFailure"))
+
+ validateCommon(config)
+ config
+ }
+
+ def validateCommon(config: CommonProducerConfig): Unit = {
+ requireNonEmpty(config.format, "--format")
+ requireNonEmpty(config.action, "--action")
+ requireNonEmpty(config.catalogName, "--catalogName")
+ if (config.retryTimes <= 0) {
+ throw new IllegalArgumentException("--retryTimes 必须大于 0")
+ }
+ }
+
+ def renderUsage(options: Seq[ActionOption], availableFormats: Seq[String]): String = {
+ val lines =
+ Seq("usage: LakehouseProducer") ++
+ mergeOptions(options).map { option =>
+ val value = if (option.hasValue) " <arg>" else ""
+ val names =
+ option.shortName
+ .map(shortName => s"-$shortName, --${option.name}$value")
+ .getOrElse(s" --${option.name}$value")
+ f" $names%-40s ${option.description}"
+ } ++
+ Seq(s"Available formats: ${availableFormats.mkString(", ")}")
+ lines.mkString(System.lineSeparator())
+ }
+
+ private def optionValue(commandLine: ProducerParsedOptions, name: String): Option[String] = {
+ Option(commandLine.getOptionValue(name)).map(_.trim).filter(_.nonEmpty)
+ }
+
+ private def stringValue(value: String): String = {
+ if (value == null) null else value.trim
+ }
+
+ private def requireNonEmpty(value: String, optionName: String): Unit = {
+ if (value == null || value.trim.isEmpty) {
+ throw new IllegalArgumentException(s"$optionName 不能为空")
+ }
+ }
+
+ private def parseInt(value: String, optionName: String): Int = {
+ try {
+ value.toInt
+ } catch {
+ case _: NumberFormatException =>
+ throw new IllegalArgumentException(s"$optionName 必须是整数: $value")
+ }
+ }
+
+ private def parseBoolean(value: String, optionName: String): Boolean = {
+ value.trim.toLowerCase(Locale.ROOT) match {
+ case "true" => true
+ case "false" => false
+ case _ => throw new IllegalArgumentException(s"$optionName 必须是 true/false: $value")
+ }
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerConfig.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerConfig.scala
new file mode 100644
index 000000000..4b5b9efcc
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerConfig.scala
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+final case class CommonProducerConfig(
+ format: String,
+ action: String,
+ catalogName: String,
+ databaseName: Option[String],
+ tableName: Option[String],
+ tableNameRegex: Option[String],
+ retryTimes: Int,
+ continueOnTableFailure: Boolean)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerContext.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerContext.scala
new file mode 100644
index 000000000..568295ca2
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerContext.scala
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.amoro.producer.spi.LakeFormatAdapter
+
+final case class ProducerContext(
+ spark: SparkSession,
+ common: CommonProducerConfig,
+ adapter: LakeFormatAdapter)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerOptions.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerOptions.scala
new file mode 100644
index 000000000..4e62ddae7
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerOptions.scala
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+final case class ActionOption(
+ name: String,
+ hasValue: Boolean = true,
+ description: String = "",
+ shortName: Option[String] = None) {
+ require(name != null && name.trim.nonEmpty, "option name must not be empty")
+ require(!name.startsWith("-"), s"option name must not start with '-': $name")
+ shortName.foreach { value =>
+ require(value.trim.nonEmpty, "short option name must not be empty")
+ require(!value.startsWith("-"), s"short option name must not start with '-': $value")
+ }
+}
+
+final case class ProducerParsedOptions private[core] (
+ values: Map[String, String],
+ flags: Set[String]) {
+
+ def getOptionValue(name: String): String = {
+ values.getOrElse(name, null)
+ }
+
+ def getOptionValue(name: String, defaultValue: String): String = {
+ values.getOrElse(name, defaultValue)
+ }
+
+ def hasOption(name: String): Boolean = {
+ values.contains(name) || flags.contains(name)
+ }
+}
+
+final class ProducerCliParseException(message: String) extends IllegalArgumentException(message)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerSummary.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerSummary.scala
new file mode 100644
index 000000000..849864034
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/ProducerSummary.scala
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+final case class FailedTableSummary(table: TargetTable, message: String)
+
+final case class ProducerSummary(
+ tableCount: Int,
+ successTables: Seq[TargetTable],
+ failedTables: Seq[TargetTable],
+ skippedTables: Seq[TargetTable],
+ failedTableSummaries: Seq[FailedTableSummary],
+ taskSuccessCount: Int,
+ taskFailedCount: Int)
+
+object ProducerSummary {
+
+ def fromResults(results: Seq[(TargetTable, ActionResult)]): ProducerSummary = {
+ val taskResults = results.flatMap(_._2.tasks)
+ ProducerSummary(
+ tableCount = results.size,
+ successTables = tablesWithStatus(results, ActionStatus.Success),
+ failedTables = tablesWithStatus(results, ActionStatus.Failed),
+ skippedTables = tablesWithStatus(results, ActionStatus.Skipped),
+ failedTableSummaries = failedTableSummaries(results),
+ taskSuccessCount = taskResults.count(_.status == ActionStatus.Success),
+ taskFailedCount = taskResults.count(_.status == ActionStatus.Failed))
+ }
+
+ private def tablesWithStatus(
+ results: Seq[(TargetTable, ActionResult)],
+ status: ActionStatus): Seq[TargetTable] = {
+ results.collect {
+ case (table, result) if result.status == status => table
+ }
+ }
+
+ private def failedTableSummaries(
+ results: Seq[(TargetTable, ActionResult)]): Seq[FailedTableSummary] = {
+ results.collect {
+ case (table, result) if result.status == ActionStatus.Failed =>
+ FailedTableSummary(table, failureMessage(result))
+ }
+ }
+
+ private def failureMessage(result: ActionResult): String = {
+ result.message
+ .orElse(
+ result.tasks
+ .find(_.status == ActionStatus.Failed)
+ .flatMap(_.message))
+ .map(normalizeMessage)
+ .filter(_.nonEmpty)
+ .getOrElse("action failed without message")
+ }
+
+ private def normalizeMessage(message: String): String = {
+ message.replaceAll("\\s+", " ").trim
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/RetryMode.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/RetryMode.scala
new file mode 100644
index 000000000..8f2e0ebf6
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/RetryMode.scala
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+sealed trait RetryMode
+
+object RetryMode {
+ case object FrameworkTableRetry extends RetryMode
+ case object ActionManagedRetry extends RetryMode
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableActionJsonLog.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableActionJsonLog.scala
new file mode 100644
index 000000000..71566bf7c
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableActionJsonLog.scala
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.util.Locale
+
+private[producer] object TableActionJsonLog {
+
+ private val EventName = "lakehouse_producer_table_action"
+ private val DeletedSnapshotsTotalKey = "deletedSnapshotsTotal"
+
+ def render(
+ common: CommonProducerConfig,
+ table: TargetTable,
+ result: ActionResult,
+ measuredCostMs: Double): String = {
+ val fields =
+ Seq(
+ jsonField("event", jsonString(EventName)),
+ jsonField("format", jsonString(common.format)),
+ jsonField("catalog", jsonString(common.catalogName)),
+ jsonField("database", jsonString(table.database)),
+ jsonField("table", jsonString(table.table)),
+ jsonField("target", jsonString(table.raw)),
+ jsonField("action", jsonString(common.action)),
+ jsonField("status", jsonString(statusName(result.status))),
+ jsonField("costMs", jsonNumber(measuredCostMs)),
+ jsonField("deletedSnapshotsTotal", deletedSnapshotsTotal(result.metrics)),
+ jsonField("metrics", jsonObject(result.metrics)),
+ jsonField("message", result.message.map(jsonStringOrNull).getOrElse("null")))
+ fields.mkString("{", ",", "}")
+ }
+
+ private def statusName(status: ActionStatus): String = {
+ status match {
+ case ActionStatus.Success => "success"
+ case ActionStatus.Failed => "failed"
+ case ActionStatus.Skipped => "skipped"
+ }
+ }
+
+ private def deletedSnapshotsTotal(metrics: Map[String, String]): String = {
+ Option(metrics)
+ .flatMap(metricsMap => metricsMap.get(DeletedSnapshotsTotalKey))
+ .flatMap(toLong)
+ .map(_.toString)
+ .getOrElse("null")
+ }
+
+ private def toLong(value: String): Option[Long] = {
+ Option(value).flatMap { nonNullValue =>
+ try {
+ Some(nonNullValue.trim.toLong)
+ } catch {
+ case _: NumberFormatException => None
+ }
+ }
+ }
+
+ private def jsonObject(values: Map[String, String]): String = {
+ Option(values).getOrElse(Map.empty).toSeq
+ .filter(_._1 != null)
+ .sortBy(_._1)
+ .map {
+ case (key, value) => s"${jsonString(key)}:${jsonStringOrNull(value)}"
+ }
+ .mkString("{", ",", "}")
+ }
+
+ private def jsonField(name: String, value: String): String = {
+ s"${jsonString(name)}:$value"
+ }
+
+ private def jsonString(value: String): String = {
+ val builder = new StringBuilder
+ builder.append('"')
+ value.foreach {
+ case '"' => builder.append("\\\"")
+ case '\\' => builder.append("\\\\")
+ case '\b' => builder.append("\\b")
+ case '\f' => builder.append("\\f")
+ case '\n' => builder.append("\\n")
+ case '\r' => builder.append("\\r")
+ case '\t' => builder.append("\\t")
+ case char if char < ' ' =>
+ builder.append("\\u")
+ builder.append(f"${char.toInt}%04x")
+ case char => builder.append(char)
+ }
+ builder.append('"')
+ builder.toString()
+ }
+
+ private def jsonStringOrNull(value: String): String = {
+ if (value == null) {
+ "null"
+ } else {
+ jsonString(value)
+ }
+ }
+
+ private def jsonNumber(value: Double): String = {
+ if (value.isNaN || value.isInfinity) {
+ "null"
+ } else {
+ java.lang.Double.toString(value).toLowerCase(Locale.ROOT)
+ }
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableFilter.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableFilter.scala
new file mode 100644
index 000000000..8e3ff0c2d
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableFilter.scala
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.util.regex.{Pattern, PatternSyntaxException}
+
+object TableFilter {
+
+ def compileTableNameRegex(regex: String): Pattern = {
+ val trimmed = Option(regex).map(_.trim).getOrElse("")
+ if (trimmed.isEmpty) {
+ throw new IllegalArgumentException("--tableNameRegex must not be empty")
+ }
+
+ try {
+ Pattern.compile(trimmed)
+ } catch {
+ case e: PatternSyntaxException =>
+ throw new IllegalArgumentException(s"Invalid --tableNameRegex: ${e.getMessage}", e)
+ }
+ }
+
+ def filterByRegex(tables: Seq[TargetTable], regex: Option[String]): Seq[TargetTable] = {
+ filterByCompiledRegex(tables, regex.map(compileTableNameRegex))
+ }
+
+ def filterByCompiledRegex(
+ tables: Seq[TargetTable],
+ pattern: Option[Pattern]): Seq[TargetTable] = {
+ pattern match {
+ case Some(pattern) =>
+ tables.filter(target => pattern.matcher(target.table).find())
+ case None =>
+ tables
+ }
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableResolver.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableResolver.scala
new file mode 100644
index 000000000..2287b3bfb
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TableResolver.scala
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.amoro.producer.spi.LakeFormatAdapter
+
+object TableResolver {
+
+ def validateTargetOptions(common: CommonProducerConfig): Unit = {
+ (common.databaseName, common.tableName) match {
+ case (Some(database), None) =>
+ requireNonEmpty(database, "--databaseName")
+ common.tableNameRegex.foreach(TableFilter.compileTableNameRegex)
+
+ case (Some(database), Some(table)) =>
+ rejectRegexWithTableName(common.tableNameRegex)
+ requireNonEmpty(database, "--databaseName")
+ requireSimpleTableName(table)
+
+ case (None, Some(table)) =>
+ rejectRegexWithTableName(common.tableNameRegex)
+ requireQualifiedTableName(table)
+
+ case (None, None) =>
+ throw new IllegalArgumentException(
+ "--databaseName or qualified --tableName must be specified")
+ }
+ }
+
+ def resolveTargets(
+ spark: SparkSession,
+ adapter: LakeFormatAdapter,
+ common: CommonProducerConfig): Seq[TargetTable] = {
+ validateTargetOptions(common)
+ (common.databaseName, common.tableName) match {
+ case (Some(database), None) =>
+ val pattern = common.tableNameRegex.map(TableFilter.compileTableNameRegex)
+ val tables = adapter.listTables(spark, common, requireNonEmpty(database, "--databaseName"))
+ TableFilter.filterByCompiledRegex(tables, pattern)
+
+ case (Some(database), Some(table)) =>
+ rejectRegexWithTableName(common.tableNameRegex)
+ val databaseName = requireNonEmpty(database, "--databaseName")
+ val tableName = requireSimpleTableName(table)
+ Seq(TargetTable(s"$databaseName.$tableName", databaseName, tableName))
+
+ case (None, Some(table)) =>
+ rejectRegexWithTableName(common.tableNameRegex)
+ val (databaseName, tableName) = requireQualifiedTableName(table)
+ Seq(TargetTable(s"$databaseName.$tableName", databaseName, tableName))
+
+ case (None, None) =>
+ throw new IllegalArgumentException(
+ "--databaseName or qualified --tableName must be specified")
+ }
+ }
+
+ private def rejectRegexWithTableName(regex: Option[String]): Unit = {
+ if (regex.isDefined) {
+ throw new IllegalArgumentException(
+ "--tableNameRegex can only be used with --databaseName without --tableName")
+ }
+ }
+
+ private def requireSimpleTableName(table: String): String = {
+ val tableName = requireNonEmpty(table, "--tableName")
+ if (tableName.contains(".")) {
+ throw new IllegalArgumentException(
+ "--tableName must be a simple table name when --databaseName is specified")
+ }
+ tableName
+ }
+
+ private def requireQualifiedTableName(table: String): (String, String) = {
+ val tableName = requireNonEmpty(table, "--tableName")
+ val parts = tableName.split("\\.", -1)
+ if (parts.length != 2 || parts.exists(_.trim.isEmpty)) {
+ throw new IllegalArgumentException(
+ "--tableName must be qualified as database.table when --databaseName is not specified")
+ }
+ (parts(0).trim, parts(1).trim)
+ }
+
+ private def requireNonEmpty(value: String, name: String): String = {
+ val trimmed = Option(value).map(_.trim).getOrElse("")
+ if (trimmed.isEmpty) {
+ throw new IllegalArgumentException(s"$name must not be empty")
+ }
+ trimmed
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TargetTable.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TargetTable.scala
new file mode 100644
index 000000000..435ae416a
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/core/TargetTable.scala
@@ -0,0 +1,21 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+final case class TargetTable(raw: String, database: String, table: String)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactAction.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactAction.scala
new file mode 100644
index 000000000..01be06306
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactAction.scala
@@ -0,0 +1,373 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import scala.util.control.NonFatal
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, ActionStatus, ActionTaskResult, CommonProducerConfig, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+import org.apache.amoro.producer.paimon.PaimonSqlUtils.{fieldOpt, quoteIdent, sqlString, valueAt}
+import org.apache.amoro.producer.spi.{ActionConfig, ProducerAction}
+
+final case class PaimonCompactConfig(
+ startBucket: Int,
+ step: Int,
+ compactStrategy: String,
+ procedureOptions: String,
+ version: String,
+ partitionIdleTime: String)
+ extends ActionConfig
+
+class PaimonCompactAction(runner: PaimonSqlRunner = SparkPaimonSqlRunner) extends ProducerAction {
+
+ override def name: String = "compact"
+
+ override def retryMode: RetryMode = RetryMode.ActionManagedRetry
+
+ override def options: Seq[ActionOption] = {
+ Seq(
+ ActionOption("startBucket", description = "Start bucket id"),
+ ActionOption("step", description = "Bucket range step"),
+ ActionOption("compactStrategy", description = "Paimon compact strategy"),
+ ActionOption("procedureOptions", description = "Paimon compact procedure options"),
+ ActionOption("version", description = "Paimon version"),
+ ActionOption("partitionIdleTime", description = "Paimon 0.9 partition idle time"))
+ }
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = {
+ PaimonCompactConfig(
+ startBucket = parseInt(optionValue(commandLine, "startBucket", "0"), "--startBucket"),
+ step = parseInt(optionValue(commandLine, "step", "20"), "--step"),
+ compactStrategy = optionValue(commandLine, "compactStrategy", "full"),
+ procedureOptions = optionValue(commandLine, "procedureOptions", "target-file-size=256m"),
+ version = optionValue(commandLine, "version", ""),
+ partitionIdleTime = optionValue(commandLine, "partitionIdleTime", "1d"))
+ }
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {
+ val config = asCompactConfig(actionConfig)
+ validateRetryTimes(common.retryTimes)
+ if (config.startBucket < 0) {
+ throw new IllegalArgumentException("--startBucket 不能小于 0")
+ }
+ if (config.step <= 0) {
+ throw new IllegalArgumentException("--step 必须大于 0")
+ }
+ if (config.compactStrategy.trim.isEmpty) {
+ throw new IllegalArgumentException("--compactStrategy 的值不能为空")
+ }
+ if (config.version.trim.nonEmpty && !PaimonCompactAction.isPaimon09(config)) {
+ throw new IllegalArgumentException("--version 目前仅支持 0.9")
+ }
+ if (PaimonCompactAction.isPaimon09(config) && config.partitionIdleTime.trim.isEmpty) {
+ throw new IllegalArgumentException("--partitionIdleTime 的值不能为空")
+ }
+ }
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ val config = asCompactConfig(actionConfig)
+ validateRetryTimes(context.common.retryTimes)
+ val startNs = System.nanoTime()
+ val bucketNumResult = readBucketNum(context, table)
+
+ val tasks: Seq[ActionTaskResult] =
+ bucketNumResult match {
+ case Left(_) =>
+ Seq(runTask(
+ context,
+ table,
+ "non-bucket",
+ PaimonCompactAction.buildNonBucketCompactSql(table)))
+
+ case Right(bucketNum) if config.startBucket > bucketNum - 1 =>
+ val reason = s"startBucket(${config.startBucket}) > 最大 bucket id(${bucketNum - 1})"
+ return result(
+ status = ActionStatus.Skipped,
+ tasks =
+ Seq(
+ ActionTaskResult(
+ name = "skipped",
+ status = ActionStatus.Skipped,
+ message = Some(reason),
+ metrics = Map("table" -> table.raw))),
+ message = Some(reason),
+ table = table,
+ startNs = startNs)
+
+ case Right(_) if PaimonCompactAction.isPaimon09(config) =>
+ val taskName = s"partition_idle_time=${config.partitionIdleTime.trim}"
+ Seq(
+ runTask(
+ context,
+ table,
+ taskName,
+ PaimonCompactAction.buildCompactSql(
+ context.common.catalogName,
+ config,
+ table,
+ bucketRange = "")))
+
+ case Right(bucketNum) =>
+ PaimonCompactAction
+ .buildBucketRanges(config.startBucket, config.step, bucketNum)
+ .map { bucketRange =>
+ runTask(
+ context,
+ table,
+ bucketRange,
+ PaimonCompactAction.buildCompactSql(
+ context.common.catalogName,
+ config,
+ table,
+ bucketRange))
+ }
+ }
+
+ val status =
+ if (tasks.exists(_.status == ActionStatus.Failed)) {
+ ActionStatus.Failed
+ } else {
+ ActionStatus.Success
+ }
+
+ result(status, tasks, None, table, startNs)
+ }
+
+ private[paimon] def buildBucketRanges(
+ startBucket: Int,
+ step: Int,
+ bucketNum: Int): Seq[String] = {
+ PaimonCompactAction.buildBucketRanges(startBucket, step, bucketNum)
+ }
+
+ private[paimon] def buildCompactSql(
+ catalogName: String,
+ config: PaimonCompactConfig,
+ table: TargetTable,
+ bucketRange: String): String = {
+ PaimonCompactAction.buildCompactSql(catalogName, config, table, bucketRange)
+ }
+
+ private[paimon] def buildNonBucketCompactSql(table: TargetTable): String = {
+ PaimonCompactAction.buildNonBucketCompactSql(table)
+ }
+
+ private[paimon] def buildNonBucketCompactSql(
+ config: PaimonCompactConfig,
+ table: TargetTable): String = {
+ PaimonCompactAction.buildNonBucketCompactSql(config, table)
+ }
+
+ private[paimon] def isPaimon09(config: PaimonCompactConfig): Boolean = {
+ PaimonCompactAction.isPaimon09(config)
+ }
+
+ private def readBucketNum(context: ProducerContext, table: TargetTable): Either[String, Int] = {
+ val optionsTable =
+ s"${quoteIdent(context.common.catalogName)}.${quoteIdent(table.database)}.${quoteIdent(table.table + "$options")}"
+ val sqlText =
+ s"""
+ |SELECT `value`
+ |FROM $optionsTable
+ |WHERE lower(trim(`key`)) = 'bucket'
+ |LIMIT 1
+ |""".stripMargin
+
+ try {
+ runner.run(context.spark, sqlText).headOption match {
+ case None =>
+ Left("no bucket option in $options")
+
+ case Some(row) =>
+ fieldOpt(row, Set("value")).orElse(valueAt(row, 0)).map(_.toString.trim) match {
+ case None =>
+ Left("bucket value is null")
+
+ case Some(bucketText) =>
+ try {
+ val bucketNum = bucketText.toInt
+ if (bucketNum <= 0) {
+ Left(s"bucket = $bucketNum (非正数,无法按 bucket 范围 compact)")
+ } else {
+ Right(bucketNum)
+ }
+ } catch {
+ case NonFatal(_) =>
+ Left(s"bucket 不是整数: $bucketText")
+ }
+ }
+ }
+ } catch {
+ case NonFatal(t) =>
+ Left(s"read bucket error: ${t.getMessage}")
+ }
+ }
+
+ private def runTask(
+ context: ProducerContext,
+ table: TargetTable,
+ taskName: String,
+ sqlText: String): ActionTaskResult = {
+ var attempt = 1
+ var lastError: Throwable = null
+
+ while (attempt <= context.common.retryTimes) {
+ try {
+ runner.run(context.spark, sqlText)
+ return ActionTaskResult(
+ name = taskName,
+ status = ActionStatus.Success,
+ message = None,
+ metrics = Map("table" -> table.raw, "attempts" -> attempt.toString))
+ } catch {
+ case NonFatal(t) =>
+ lastError = t
+ attempt += 1
+ }
+ }
+
+ ActionTaskResult(
+ name = taskName,
+ status = ActionStatus.Failed,
+ message = Some(errorMessage(lastError)),
+ metrics = Map("table" -> table.raw, "attempts" -> context.common.retryTimes.toString))
+ }
+
+ private def result(
+ status: ActionStatus,
+ tasks: Seq[ActionTaskResult],
+ message: Option[String],
+ table: TargetTable,
+ startNs: Long): ActionResult = {
+ ActionResult(
+ status = status,
+ tasks = tasks,
+ message = message,
+ metrics = Map("table" -> table.raw),
+ costMs = (System.nanoTime() - startNs) / 1e6)
+ }
+
+ private def asCompactConfig(actionConfig: ActionConfig): PaimonCompactConfig = {
+ actionConfig match {
+ case config: PaimonCompactConfig => config
+ case other =>
+ throw new IllegalArgumentException(
+ s"Action compact requires PaimonCompactConfig, but got ${configType(other)}")
+ }
+ }
+
+ private def configType(actionConfig: ActionConfig): String = {
+ Option(actionConfig).map(_.getClass.getName).getOrElse("null")
+ }
+
+ private def parseInt(value: String, optionName: String): Int = {
+ try {
+ value.trim.toInt
+ } catch {
+ case _: NumberFormatException =>
+ throw new IllegalArgumentException(s"$optionName 必须是整数: $value")
+ }
+ }
+
+ private def optionValue(
+ commandLine: ProducerParsedOptions,
+ optionName: String,
+ defaultValue: String): String = {
+ commandLine.getOptionValue(optionName, defaultValue).trim
+ }
+
+ private def validateRetryTimes(retryTimes: Int): Unit = {
+ if (retryTimes <= 0) {
+ throw new IllegalArgumentException("--retryTimes 必须大于 0")
+ }
+ }
+
+ private def errorMessage(error: Throwable): String = {
+ Option(error)
+ .map(t => s"${t.getClass.getSimpleName}: ${Option(t.getMessage).getOrElse("")}")
+ .getOrElse("unknown error")
+ }
+}
+
+object PaimonCompactAction {
+
+ private[paimon] def buildBucketRanges(
+ startBucket: Int,
+ step: Int,
+ bucketNum: Int): Seq[String] = {
+ var start = startBucket
+ var ranges = Seq.empty[String]
+ while (start < bucketNum) {
+ val end = math.min(start + step - 1, bucketNum - 1)
+ ranges :+= s"$start-$end"
+ start = end + 1
+ }
+ ranges
+ }
+
+ private[paimon] def buildCompactSql(
+ catalogName: String,
+ config: PaimonCompactConfig,
+ table: TargetTable,
+ bucketRange: String): String = {
+ if (isPaimon09(config)) {
+ return s"""
+ |CALL sys.compact(
+ | table => ${sqlString(table.raw)},
+ | partition_idle_time => ${sqlString(config.partitionIdleTime.trim)}
+ |)
+ |""".stripMargin
+ }
+
+ val optionsClause =
+ if (config.procedureOptions.trim.isEmpty) {
+ ""
+ } else {
+ s",\n options => ${sqlString(config.procedureOptions.trim)}"
+ }
+
+ s"""
+ |CALL ${quoteIdent(catalogName)}.sys.compact(
+ | table => ${sqlString(table.raw)},
+ | buckets => ${sqlString(bucketRange)},
+ | compact_strategy => ${sqlString(config.compactStrategy.trim)}$optionsClause
+ |)
+ |""".stripMargin
+ }
+
+ private[paimon] def buildNonBucketCompactSql(table: TargetTable): String = {
+ s"""
+ |CALL sys.compact(
+ | table => ${sqlString(table.raw)}
+ |)
+ |""".stripMargin
+ }
+
+ private[paimon] def buildNonBucketCompactSql(
+ config: PaimonCompactConfig,
+ table: TargetTable): String = {
+ buildNonBucketCompactSql(table)
+ }
+
+ private[paimon] def isPaimon09(config: PaimonCompactConfig): Boolean = {
+ config.version.trim == "0.9"
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestAction.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestAction.scala
new file mode 100644
index 000000000..1a9d2c193
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestAction.scala
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, ActionStatus, ActionTaskResult, CommonProducerConfig, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+import org.apache.amoro.producer.paimon.PaimonSqlUtils.{fieldOpt, quoteIdent, sqlString, valueAt}
+import org.apache.amoro.producer.spi.{ActionConfig, ProducerAction}
+
+final case class PaimonCompactManifestConfig(procedureOptions: String) extends ActionConfig
+
+class PaimonCompactManifestAction(runner: PaimonSqlRunner = SparkPaimonSqlRunner)
+ extends ProducerAction {
+
+ override def name: String = "compact-manifest"
+
+ override def retryMode: RetryMode = RetryMode.FrameworkTableRetry
+
+ override def options: Seq[ActionOption] = {
+ Seq(ActionOption("procedureOptions", description = "Paimon compact_manifest procedure options"))
+ }
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = {
+ PaimonCompactManifestConfig(procedureOptions =
+ commandLine.getOptionValue("procedureOptions", "").trim)
+ }
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {
+ asCompactManifestConfig(actionConfig)
+ }
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ val config = asCompactManifestConfig(actionConfig)
+ val startNs = System.nanoTime()
+ val sqlText =
+ PaimonCompactManifestAction.buildCompactManifestSql(
+ context.common.catalogName,
+ config,
+ table)
+ val rows = runner.run(context.spark, sqlText)
+ val result = parseResult(rows)
+ val metrics = Map("table" -> table.raw, "result" -> result.toString)
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks =
+ Seq(
+ ActionTaskResult(
+ name = "compact_manifest",
+ status = ActionStatus.Success,
+ message = None,
+ metrics = metrics)),
+ message = None,
+ metrics = metrics,
+ costMs = (System.nanoTime() - startNs) / 1e6)
+ }
+
+ private def asCompactManifestConfig(actionConfig: ActionConfig): PaimonCompactManifestConfig = {
+ actionConfig match {
+ case config: PaimonCompactManifestConfig => config
+ case other =>
+ throw new IllegalArgumentException(
+ s"Action compact-manifest requires PaimonCompactManifestConfig, but got ${configType(other)}")
+ }
+ }
+
+ private def configType(actionConfig: ActionConfig): String = {
+ Option(actionConfig).map(_.getClass.getName).getOrElse("null")
+ }
+
+ private def parseResult(rows: Seq[org.apache.spark.sql.Row]): Boolean = {
+ rows.headOption
+ .flatMap(row => fieldOpt(row, Set("result")).orElse(valueAt(row, 0)))
+ .exists(toBoolean)
+ }
+
+ private def toBoolean(value: Any): Boolean = {
+ value match {
+ case null => false
+ case b: java.lang.Boolean => b.booleanValue()
+ case s: String => s.trim.equalsIgnoreCase("true") || s.trim == "1"
+ case i: java.lang.Integer => i == 1
+ case l: java.lang.Long => l == 1L
+ case other => other.toString.trim.equalsIgnoreCase("true")
+ }
+ }
+}
+
+object PaimonCompactManifestAction {
+
+ private[paimon] def buildCompactManifestSql(
+ catalogName: String,
+ config: PaimonCompactManifestConfig,
+ table: TargetTable): String = {
+ val optionsClause =
+ if (config.procedureOptions.trim.isEmpty) {
+ ""
+ } else {
+ s",\n options => ${sqlString(config.procedureOptions.trim)}"
+ }
+
+ s"""
+ |CALL ${quoteIdent(catalogName)}.sys.compact_manifest(
+ | table => ${sqlString(table.raw)}$optionsClause
+ |)
+ |""".stripMargin
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsAction.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsAction.scala
new file mode 100644
index 000000000..f5126a70d
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsAction.scala
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.sql.Row
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, ActionStatus, ActionTaskResult, CommonProducerConfig, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+import org.apache.amoro.producer.paimon.PaimonSqlUtils.{fieldOpt, quoteIdent, sqlString, valueAt}
+import org.apache.amoro.producer.spi.{ActionConfig, ProducerAction}
+
+final case class PaimonExpireSnapshotsConfig(
+ retainMax: Int,
+ retainMin: Int,
+ maxDeletes: Long,
+ procedureOptions: String)
+ extends ActionConfig
+
+class PaimonExpireSnapshotsAction(runner: PaimonSqlRunner = SparkPaimonSqlRunner)
+ extends ProducerAction {
+
+ override def name: String = "expire-snapshots"
+
+ override def retryMode: RetryMode = RetryMode.ActionManagedRetry
+
+ override def options: Seq[ActionOption] = {
+ Seq(
+ ActionOption("retainMax", description = "Paimon expire_snapshots retain max"),
+ ActionOption("retainMin", description = "Paimon expire_snapshots retain min"),
+ ActionOption("maxDeletes", description = "Paimon expire_snapshots max deletes"),
+ ActionOption("procedureOptions", description = "Paimon expire_snapshots procedure options"))
+ }
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = {
+ PaimonExpireSnapshotsConfig(
+ retainMax =
+ parseInt(
+ commandLine.getOptionValue(
+ "retainMax",
+ PaimonExpireSnapshotsAction.DefaultRetainMax.toString),
+ "--retainMax"),
+ retainMin =
+ parseInt(
+ commandLine.getOptionValue(
+ "retainMin",
+ PaimonExpireSnapshotsAction.DefaultRetainMin.toString),
+ "--retainMin"),
+ maxDeletes =
+ parseLong(
+ commandLine.getOptionValue(
+ "maxDeletes",
+ PaimonExpireSnapshotsAction.DefaultMaxDeletes.toString),
+ "--maxDeletes"),
+ procedureOptions =
+ commandLine
+ .getOptionValue(
+ "procedureOptions",
+ PaimonExpireSnapshotsAction.DefaultProcedureOptions)
+ .trim)
+ }
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {
+ val config = asExpireSnapshotsConfig(actionConfig)
+ validateRetryTimes(common.retryTimes)
+ if (config.retainMax < 0) {
+ throw new IllegalArgumentException("--retainMax 不能小于 0")
+ }
+ if (config.retainMin < 0) {
+ throw new IllegalArgumentException("--retainMin 不能小于 0")
+ }
+ if (config.maxDeletes <= 0) {
+ throw new IllegalArgumentException("--maxDeletes 必须大于 0")
+ }
+ }
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ validate(context.common, actionConfig)
+ val config = asExpireSnapshotsConfig(actionConfig)
+ val startNs = System.nanoTime()
+ var round = 1
+ var shouldContinue = true
+ var tasks = Seq.empty[ActionTaskResult]
+ var totalDeletedCount = 0L
+
+ while (shouldContinue) {
+ val (deletedCount, attempts) = runRoundWithRetry(context, table, config, round)
+ totalDeletedCount += deletedCount
+ tasks :+= ActionTaskResult(
+ name = s"round-$round",
+ status = ActionStatus.Success,
+ message = None,
+ metrics =
+ Map(
+ "table" -> table.raw,
+ "deletedCount" -> deletedCount.toString,
+ "attempts" -> attempts.toString))
+ shouldContinue = deletedCount >= config.maxDeletes
+ round += 1
+ }
+
+ val metrics =
+ Map(
+ "table" -> table.raw,
+ "rounds" -> tasks.size.toString,
+ "totalDeletedCount" -> totalDeletedCount.toString,
+ "deletedSnapshotsTotal" -> totalDeletedCount.toString,
+ "maxDeletes" -> config.maxDeletes.toString)
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks = tasks,
+ message = None,
+ metrics = metrics,
+ costMs = (System.nanoTime() - startNs) / 1e6)
+ }
+
+ private def runRoundWithRetry(
+ context: ProducerContext,
+ table: TargetTable,
+ config: PaimonExpireSnapshotsConfig,
+ round: Int): (Long, Int) = {
+ val sqlText =
+ PaimonExpireSnapshotsAction.buildExpireSnapshotsSql(
+ context.common.catalogName,
+ config,
+ table)
+ var attempt = 1
+ var lastError: Throwable = null
+
+ while (attempt <= context.common.retryTimes) {
+ try {
+ val deletedCount = parseDeletedCount(runner.run(context.spark, sqlText))
+ return (deletedCount, attempt)
+ } catch {
+ case NonFatal(t) =>
+ lastError = t
+ if (attempt == context.common.retryTimes) {
+ throw new RuntimeException(
+ s"表 ${table.raw} expire_snapshots 第 $round 轮连续失败 ${context.common.retryTimes} 次",
+ lastError)
+ }
+ attempt += 1
+ }
+ }
+ throw new RuntimeException("不应到达此分支")
+ }
+
+ private def parseDeletedCount(rows: Seq[Row]): Long = {
+ rows.headOption.map(parseDeletedCount).getOrElse {
+ throw new IllegalStateException("expire_snapshots 返回空结果")
+ }
+ }
+
+ private def parseDeletedCount(row: Row): Long = {
+ val deletedCount =
+ fieldOpt(row, Set("deletedcount", "deletedsnapshots", "deletedsnapshotscount", "result"))
+ .orElse(valueAt(row, 0))
+
+ deletedCount.flatMap(toLongOpt).getOrElse {
+ throw new IllegalStateException(
+ s"无法从返回结果解析删除数量,row=${row.mkString("[", ", ", "]")}")
+ }
+ }
+
+ private def asExpireSnapshotsConfig(actionConfig: ActionConfig): PaimonExpireSnapshotsConfig = {
+ actionConfig match {
+ case config: PaimonExpireSnapshotsConfig => config
+ case other =>
+ throw new IllegalArgumentException(
+ s"Action expire-snapshots requires PaimonExpireSnapshotsConfig, but got ${configType(other)}")
+ }
+ }
+
+ private def configType(actionConfig: ActionConfig): String = {
+ Option(actionConfig).map(_.getClass.getName).getOrElse("null")
+ }
+
+ private def parseInt(value: String, optionName: String): Int = {
+ try {
+ value.trim.toInt
+ } catch {
+ case _: NumberFormatException =>
+ throw new IllegalArgumentException(s"$optionName 必须是整数: $value")
+ }
+ }
+
+ private def parseLong(value: String, optionName: String): Long = {
+ try {
+ value.trim.toLong
+ } catch {
+ case _: NumberFormatException =>
+ throw new IllegalArgumentException(s"$optionName 必须是长整数: $value")
+ }
+ }
+
+ private def validateRetryTimes(retryTimes: Int): Unit = {
+ if (retryTimes <= 0) {
+ throw new IllegalArgumentException("--retryTimes 必须大于 0")
+ }
+ }
+
+ private def toLongOpt(value: Any): Option[Long] = {
+ value match {
+ case null => None
+ case n: java.lang.Number => Some(n.longValue())
+ case s: String if s.trim.matches("-?\\d+") => Some(s.trim.toLong)
+ case _ => None
+ }
+ }
+
+}
+
+object PaimonExpireSnapshotsAction {
+
+ private val DefaultRetainMax = 20
+ private val DefaultRetainMin = 10
+ private val DefaultMaxDeletes = 550L
+ private val DefaultProcedureOptions =
+ "file-operation.thread-num=32,snapshot.expire.execution-mode=sync"
+
+ private[paimon] def buildExpireSnapshotsSql(
+ catalogName: String,
+ config: PaimonExpireSnapshotsConfig,
+ table: TargetTable): String = {
+ val optionsClause =
+ if (config.procedureOptions.trim.isEmpty) {
+ ""
+ } else {
+ s",\n options => ${sqlString(config.procedureOptions.trim)}"
+ }
+
+ s"""
+ |CALL ${quoteIdent(catalogName)}.sys.expire_snapshots(
+ | table => ${sqlString(table.raw)},
+ | retain_max => ${config.retainMax},
+ | retain_min => ${config.retainMin},
+ | max_deletes => ${config.maxDeletes}$optionsClause
+ |)
+ |""".stripMargin
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapter.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapter.scala
new file mode 100644
index 000000000..b1c1ffc2d
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapter.scala
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+
+import org.apache.amoro.producer.core.{CommonProducerConfig, TargetTable}
+import org.apache.amoro.producer.paimon.PaimonSqlUtils.{fieldOpt, quoteIdent, valueAt}
+import org.apache.amoro.producer.spi.{LakeFormatAdapter, ProducerAction}
+
+class PaimonFormatAdapter(runner: PaimonSqlRunner = SparkPaimonSqlRunner)
+ extends LakeFormatAdapter {
+
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = {
+ Seq(
+ new PaimonCompactAction(runner),
+ new PaimonCompactManifestAction(runner),
+ new PaimonExpireSnapshotsAction(runner),
+ new PaimonRemoveOrphanFilesAction(runner))
+ }
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = {
+ val showTablesSql =
+ s"SHOW TABLES IN ${quoteIdent(config.catalogName)}.${quoteIdent(database)}"
+
+ runner.run(spark, showTablesSql).flatMap { row =>
+ tableName(row).filter(_.nonEmpty).filter(_ => !isTemporary(row)).map { table =>
+ TargetTable(s"$database.$table", database, table)
+ }
+ }
+ }
+
+ private def tableName(row: Row): Option[String] = {
+ fieldOpt(row, Set("tablename", "table"))
+ .orElse(valueAt(row, 1))
+ .filter(_ != null)
+ .map(_.toString.trim)
+ }
+
+ private def isTemporary(row: Row): Boolean = {
+ fieldOpt(row, Set("istemporary", "temporary"))
+ .orElse(valueAt(row, 2))
+ .exists(toBoolean)
+ }
+
+ private def toBoolean(value: Any): Boolean = {
+ value match {
+ case null => false
+ case b: java.lang.Boolean => b.booleanValue()
+ case s: String => s.trim.equalsIgnoreCase("true") || s.trim == "1"
+ case i: java.lang.Integer => i == 1
+ case l: java.lang.Long => l == 1L
+ case other => other.toString.trim.equalsIgnoreCase("true")
+ }
+ }
+}
+
+object PaimonFormatAdapter extends PaimonFormatAdapter(SparkPaimonSqlRunner)
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesAction.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesAction.scala
new file mode 100644
index 000000000..ced3d6274
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesAction.scala
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, ActionStatus, ActionTaskResult, CommonProducerConfig, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+import org.apache.amoro.producer.paimon.PaimonSqlUtils.{fieldOpt, quoteIdent, sqlString, valueAt}
+import org.apache.amoro.producer.spi.{ActionConfig, ProducerAction}
+
+final case class PaimonRemoveOrphanFilesConfig(parallelism: Int, mode: String)
+ extends ActionConfig
+
+class PaimonRemoveOrphanFilesAction(runner: PaimonSqlRunner = SparkPaimonSqlRunner)
+ extends ProducerAction {
+
+ override def name: String = "remove-orphan-files"
+
+ override def retryMode: RetryMode = RetryMode.FrameworkTableRetry
+
+ override def options: Seq[ActionOption] = {
+ Seq(
+ ActionOption("parallelism", description = "Paimon remove_orphan_files parallelism"),
+ ActionOption("mode", description = "Paimon remove_orphan_files mode"))
+ }
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = {
+ PaimonRemoveOrphanFilesConfig(
+ parallelism = parseInt(commandLine.getOptionValue("parallelism", "10"), "--parallelism"),
+ mode = commandLine.getOptionValue("mode", "distributed").trim)
+ }
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {
+ val config = asRemoveOrphanFilesConfig(actionConfig)
+ if (config.parallelism <= 0) {
+ throw new IllegalArgumentException("--parallelism 必须大于 0")
+ }
+ if (config.mode.trim.isEmpty) {
+ throw new IllegalArgumentException("--mode 的值不能为空")
+ }
+ }
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ val config = asRemoveOrphanFilesConfig(actionConfig)
+ val startNs = System.nanoTime()
+ val sqlText =
+ PaimonRemoveOrphanFilesAction.buildRemoveOrphanFilesSql(
+ context.common.catalogName,
+ config,
+ table)
+ val (deletedFileCount, deletedFileTotalLenInBytes) = parseMetrics(
+ runner.run(context.spark, sqlText))
+ val metrics =
+ Map(
+ "table" -> table.raw,
+ "deletedFileCount" -> deletedFileCount.toString,
+ "deletedFileTotalLenInBytes" -> deletedFileTotalLenInBytes.toString)
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks =
+ Seq(
+ ActionTaskResult(
+ name = "remove_orphan_files",
+ status = ActionStatus.Success,
+ message = None,
+ metrics = metrics)),
+ message = None,
+ metrics = metrics,
+ costMs = (System.nanoTime() - startNs) / 1e6)
+ }
+
+ private def asRemoveOrphanFilesConfig(actionConfig: ActionConfig)
+ : PaimonRemoveOrphanFilesConfig = {
+ actionConfig match {
+ case config: PaimonRemoveOrphanFilesConfig => config
+ case other =>
+ throw new IllegalArgumentException(
+ s"Action remove-orphan-files requires PaimonRemoveOrphanFilesConfig, but got ${configType(other)}")
+ }
+ }
+
+ private def configType(actionConfig: ActionConfig): String = {
+ Option(actionConfig).map(_.getClass.getName).getOrElse("null")
+ }
+
+ private def parseMetrics(rows: Seq[org.apache.spark.sql.Row]): (Long, Long) = {
+ var deletedFileCount = 0L
+ var deletedFileTotalLenInBytes = 0L
+
+ rows.foreach { row =>
+ val countValue =
+ fieldOpt(row, Set("deletedfilecount")).orElse(valueAt(row, 0)).getOrElse(0L)
+ val bytesValue =
+ fieldOpt(row, Set("deletedfiletotalleninbytes")).orElse(valueAt(row, 1)).getOrElse(0L)
+
+ deletedFileCount += toLong(countValue)
+ deletedFileTotalLenInBytes += toLong(bytesValue)
+ }
+
+ (deletedFileCount, deletedFileTotalLenInBytes)
+ }
+
+ private def parseInt(value: String, optionName: String): Int = {
+ try {
+ value.trim.toInt
+ } catch {
+ case _: NumberFormatException =>
+ throw new IllegalArgumentException(s"$optionName 必须是整数: $value")
+ }
+ }
+
+ private def toLong(value: Any): Long = {
+ value match {
+ case null => 0L
+ case n: java.lang.Number => n.longValue()
+ case s: String =>
+ val trimmed = s.trim
+ if (trimmed.isEmpty) {
+ 0L
+ } else {
+ trimmed.toLong
+ }
+ case other => other.toString.toLong
+ }
+ }
+}
+
+object PaimonRemoveOrphanFilesAction {
+
+ private[paimon] def buildRemoveOrphanFilesSql(
+ catalogName: String,
+ config: PaimonRemoveOrphanFilesConfig,
+ table: TargetTable): String = {
+ s"""
+ |CALL ${quoteIdent(catalogName)}.sys.remove_orphan_files(
+ | table => ${sqlString(table.raw)},
+ | parallelism => ${config.parallelism},
+ | mode => ${sqlString(config.mode.trim)}
+ |)
+ |""".stripMargin
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlRunner.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlRunner.scala
new file mode 100644
index 000000000..58f8b6b47
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlRunner.scala
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+
+trait PaimonSqlRunner {
+
+ def run(spark: SparkSession, sqlText: String): Seq[Row]
+}
+
+object SparkPaimonSqlRunner extends PaimonSqlRunner {
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ spark.sql(sqlText).collect().toSeq
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlUtils.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlUtils.scala
new file mode 100644
index 000000000..cd50205c7
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/paimon/PaimonSqlUtils.scala
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import java.util.Locale
+
+import scala.util.control.NonFatal
+
+import org.apache.spark.sql.Row
+
+private[paimon] object PaimonSqlUtils {
+
+ def quoteIdent(name: String): String = {
+ "`" + name.replace("`", "``") + "`"
+ }
+
+ def sqlString(value: String): String = {
+ "'" + value.replace("'", "''") + "'"
+ }
+
+ def fieldOpt(row: Row, normalizedNames: Set[String]): Option[Any] = {
+ val schemaOpt =
+ try {
+ Option(row.schema)
+ } catch {
+ case NonFatal(_) => None
+ }
+
+ schemaOpt.flatMap { schema =>
+ schema.fields.zipWithIndex
+ .find { case (field, _) => normalizedNames.contains(normalizeFieldName(field.name)) }
+ .flatMap { case (_, index) => valueAt(row, index) }
+ }
+ }
+
+ def valueAt(row: Row, index: Int): Option[Any] = {
+ if (index >= 0 && index < row.size && !row.isNullAt(index)) {
+ Some(row.get(index))
+ } else {
+ None
+ }
+ }
+
+ def normalizeFieldName(name: String): String = {
+ name.toLowerCase(Locale.ROOT).replace("_", "")
+ }
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/LakeFormatAdapter.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/LakeFormatAdapter.scala
new file mode 100644
index 000000000..f8dc1865e
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/LakeFormatAdapter.scala
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.spi
+
+import org.apache.spark.sql.SparkSession
+
+import org.apache.amoro.producer.core.{CommonProducerConfig, TargetTable}
+
+trait LakeFormatAdapter {
+
+ def format: String
+
+ def actions: Seq[ProducerAction]
+
+ def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable]
+}
diff --git a/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/ProducerAction.scala b/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/ProducerAction.scala
new file mode 100644
index 000000000..e7d1b9cce
--- /dev/null
+++ b/amoro-producer/src/main/scala/org/apache/amoro/producer/spi/ProducerAction.scala
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.spi
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, CommonProducerConfig, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+
+trait ActionConfig
+
+trait ProducerAction {
+
+ def name: String
+
+ def retryMode: RetryMode
+
+ def options: Seq[ActionOption]
+
+ def parse(commandLine: ProducerParsedOptions): ActionConfig
+
+ def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit
+
+ def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/LakehouseProducerTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/LakehouseProducerTest.scala
new file mode 100644
index 000000000..a25df6fe8
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/LakehouseProducerTest.scala
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer
+
+import java.io.{ByteArrayOutputStream, PrintStream}
+
+import org.apache.spark.sql.SparkSession
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertNull, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{ActionOption, ActionResult, ActionStatus, ActionTaskResult, CommonProducerConfig, ProducerApp, ProducerContext, ProducerParsedOptions, RetryMode, TargetTable}
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class LakehouseProducerTest {
+
+ @Test
+ def helpReturnsZeroAndDoesNotCreateSpark(): Unit = {
+ var sparkCreated = false
+
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--help"),
+ Seq(new FakeAdapter("paimon", new FakeAction("compact"))),
+ () => {
+ sparkCreated = true
+ null
+ })
+
+ assertEquals(0, exitCode)
+ assertFalse(sparkCreated)
+ }
+
+ @Test
+ def missingFormatOrActionReturnsTwoAndDoesNotCreateSpark(): Unit = {
+ var sparkCreated = false
+ val adapter = new FakeAdapter("paimon", new FakeAction("compact"))
+ val builder = () => {
+ sparkCreated = true
+ null
+ }
+
+ assertEquals(2, ProducerApp.runInternal(Array("--action", "compact"), Seq(adapter), builder))
+ assertEquals(2, ProducerApp.runInternal(Array("--format", "paimon"), Seq(adapter), builder))
+ assertFalse(sparkCreated)
+ }
+
+ @Test
+ def invalidTargetOptionsReturnTwoAndDoNotCreateSpark(): Unit = {
+ var sparkCreated = false
+ val adapter = new FakeAdapter("paimon", new FakeAction("compact"))
+ val builder = () => {
+ sparkCreated = true
+ null
+ }
+
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact"),
+ Seq(adapter),
+ builder))
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact", "--tableName", "t1"),
+ Seq(adapter),
+ builder))
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array(
+ "--format",
+ "paimon",
+ "--action",
+ "compact",
+ "--databaseName",
+ "db",
+ "--tableNameRegex",
+ "["),
+ Seq(adapter),
+ builder))
+ assertFalse(sparkCreated)
+ }
+
+ @Test
+ def unsupportedFormatOrActionReturnsTwoAndDoesNotCreateSpark(): Unit = {
+ var sparkCreated = false
+ val adapter = new FakeAdapter("paimon", new FakeAction("compact"))
+ val builder = () => {
+ sparkCreated = true
+ null
+ }
+
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "iceberg", "--action", "compact"),
+ Seq(adapter),
+ builder))
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "missing"),
+ Seq(adapter),
+ builder))
+ assertFalse(sparkCreated)
+ }
+
+ @Test
+ def sparkBuilderIllegalArgumentExceptionReturnsOne(): Unit = {
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact", "--databaseName", "db"),
+ Seq(new FakeAdapter("paimon", new FakeAction("compact"))),
+ () => throw new IllegalArgumentException("builder failed"))
+
+ assertEquals(1, exitCode)
+ }
+
+ @Test
+ def actionRuntimeIllegalArgumentExceptionReturnsOne(): Unit = {
+ val exitCode =
+ ProducerApp.runInternal(
+ Array(
+ "--format",
+ "paimon",
+ "--action",
+ "compact",
+ "--databaseName",
+ "db",
+ "--continueOnTableFailure",
+ "false"),
+ Seq(new FakeAdapter("paimon", new FakeAction("compact", failExecution = true))),
+ () => null)
+
+ assertEquals(1, exitCode)
+ }
+
+ @Test
+ def actionReturnedFailedStatusReturnsZeroWhenContinueTrue(): Unit = {
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact", "--databaseName", "db"),
+ Seq(
+ new FakeAdapter(
+ "paimon",
+ new FakeAction(
+ "compact",
+ resultStatusByTable = Map("db.t1" -> ActionStatus.Failed)))),
+ () => null)
+
+ assertEquals(0, exitCode)
+ }
+
+ @Test
+ def partialReturnedFailedStatusReturnsZeroAndContinuesWhenConfigured(): Unit = {
+ val action =
+ new FakeAction(
+ "compact",
+ resultStatusByTable = Map("db.t1" -> ActionStatus.Failed))
+ val adapter =
+ new FakeAdapter(
+ "paimon",
+ action,
+ tables = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")))
+
+ val (output, exitCode) =
+ captureStdout {
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact", "--databaseName", "db"),
+ Seq(adapter),
+ () => null)
+ }
+
+ assertEquals(0, exitCode)
+ assertTrue(output.contains("failed tables:"))
+ assertTrue(output.contains(" - db.t1 | failed table db.t1"))
+ assertEquals(
+ Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")),
+ action.executedTables)
+ }
+
+ @Test
+ def returnedFailedStatusStopsAndReturnsOneWhenContinueFalse(): Unit = {
+ val action =
+ new FakeAction(
+ "compact",
+ resultStatusByTable = Map("db.t1" -> ActionStatus.Failed))
+ val adapter =
+ new FakeAdapter(
+ "paimon",
+ action,
+ tables = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")))
+
+ val exitCode =
+ ProducerApp.runInternal(
+ Array(
+ "--format",
+ "paimon",
+ "--action",
+ "compact",
+ "--databaseName",
+ "db",
+ "--continueOnTableFailure",
+ "false"),
+ Seq(adapter),
+ () => null)
+
+ assertEquals(1, exitCode)
+ assertEquals(Seq(TargetTable("db.t1", "db", "t1")), action.executedTables)
+ }
+
+ @Test
+ def runInternalPrintsTableActionJsonForEachTableAndKeepsSummary(): Unit = {
+ val action =
+ new FakeAction(
+ "expire-snapshots",
+ metricsByTable =
+ Map(
+ "db.t1" -> Map("deletedSnapshotsTotal" -> "3", "rounds" -> "1"),
+ "db.t2" -> Map("deletedSnapshotsTotal" -> "5", "rounds" -> "2")))
+ val adapter =
+ new FakeAdapter(
+ "paimon",
+ action,
+ tables = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")))
+
+ val (output, exitCode) =
+ captureStdout {
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "expire-snapshots", "--databaseName", "db"),
+ Seq(adapter),
+ () => null)
+ }
+
+ assertEquals(0, exitCode)
+ val jsonLines = tableActionJsonLines(output)
+ assertEquals(2, jsonLines.size)
+ assertTrue(jsonLines.head.contains("\"action\":\"expire-snapshots\""))
+ assertTrue(jsonLines.head.contains("\"target\":\"db.t1\""))
+ assertTrue(jsonLines.head.contains("\"deletedSnapshotsTotal\":3"))
+ assertTrue(jsonLines(1).contains("\"target\":\"db.t2\""))
+ assertTrue(jsonLines(1).contains("\"deletedSnapshotsTotal\":5"))
+ assertTrue(output.contains("=========== PRODUCER SUMMARY ==========="))
+ assertEquals(
+ Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")),
+ action.executedTables)
+ }
+
+ @Test
+ def resolveAdapterAndActionSuccessfullyBeforeValidation(): Unit = {
+ var sparkCreated = false
+ val action = new FakeAction("compact", failValidation = true)
+ val adapter = new FakeAdapter("paimon", action)
+
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--format", "PAIMON", "--action", "COMPACT"),
+ Seq(adapter),
+ () => {
+ sparkCreated = true
+ null
+ })
+
+ assertEquals(2, exitCode)
+ assertEquals(1, action.parseCalls)
+ assertEquals(1, action.validateCalls)
+ assertFalse(sparkCreated)
+ }
+
+ @Test
+ def runInternalResolvesTargetsAndExecutesTables(): Unit = {
+ val action = new FakeAction("compact")
+ val adapter = new FakeAdapter("paimon", action)
+ var sparkCreated = false
+
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "compact", "--databaseName", "db"),
+ Seq(adapter),
+ () => {
+ sparkCreated = true
+ null
+ })
+
+ assertEquals(0, exitCode)
+ assertTrue(sparkCreated)
+ assertEquals(Seq("db"), adapter.listedDatabases)
+ assertEquals(Seq(TargetTable("db.t1", "db", "t1")), action.executedTables)
+ assertNull(action.contextSpark)
+ }
+
+ @Test
+ def lakehouseProducerUsesPaimonAdapter(): Unit = {
+ assertEquals(Seq("paimon"), LakehouseProducer.adapters.map(_.format))
+ }
+
+ private object NoopConfig extends ActionConfig
+
+ private def captureStdout[T](block: => T): (String, T) = {
+ val output = new ByteArrayOutputStream()
+ val printStream = new PrintStream(output, true, "UTF-8")
+ val result = Console.withOut(printStream) {
+ block
+ }
+ printStream.flush()
+ output.toString("UTF-8") -> result
+ }
+
+ private def tableActionJsonLines(output: String): Seq[String] = {
+ output.linesIterator.filter(_.contains("\"event\":\"lakehouse_producer_table_action\"")).toSeq
+ }
+
+ final private class FakeAdapter(
+ formatName: String,
+ action: ProducerAction,
+ tables: Seq[TargetTable] = Seq(TargetTable("db.t1", "db", "t1")))
+ extends LakeFormatAdapter {
+ var listedDatabases: Seq[String] = Seq.empty
+
+ override def format: String = formatName
+
+ override def actions: Seq[ProducerAction] = Seq(action)
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = {
+ listedDatabases = listedDatabases :+ database
+ tables.map(table => table.copy(raw = s"$database.${table.table}", database = database))
+ }
+ }
+
+ final private class FakeAction(
+ actionName: String,
+ failValidation: Boolean = false,
+ failExecution: Boolean = false,
+ resultStatusByTable: Map[String, ActionStatus] = Map.empty,
+ metricsByTable: Map[String, Map[String, String]] = Map.empty)
+ extends ProducerAction {
+ var parseCalls = 0
+ var validateCalls = 0
+ var executedTables: Seq[TargetTable] = Seq.empty
+ var contextSpark: SparkSession = _
+
+ override def name: String = actionName
+
+ override def retryMode: RetryMode = RetryMode.ActionManagedRetry
+
+ override def options: Seq[ActionOption] = Seq.empty
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = {
+ parseCalls += 1
+ NoopConfig
+ }
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {
+ validateCalls += 1
+ if (failValidation) {
+ throw new IllegalArgumentException("validation failed")
+ }
+ }
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ contextSpark = context.spark
+ executedTables = executedTables :+ table
+ if (failExecution) {
+ throw new IllegalArgumentException("execution failed")
+ }
+ val status = resultStatusByTable.getOrElse(table.raw, ActionStatus.Success)
+ ActionResult(
+ status = status,
+ tasks = Seq(ActionTaskResult(table.raw, status, None, Map.empty)),
+ message = if (status == ActionStatus.Failed) Some(s"failed table ${table.raw}") else None,
+ metrics = metricsByTable.getOrElse(table.raw, Map.empty) + ("table" -> table.raw),
+ costMs = 0.0d)
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ActionResultTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ActionResultTest.scala
new file mode 100644
index 000000000..9c5112772
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ActionResultTest.scala
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import org.apache.spark.sql.SparkSession
+import org.junit.jupiter.api.Assertions.assertEquals
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class ActionResultTest {
+
+ @Test
+ def taskResultUsesSpecDefinedFields(): Unit = {
+ val task =
+ ActionTaskResult(
+ name = "db.tbl",
+ status = ActionStatus.Success,
+ message = Some("compact finished"),
+ metrics = Map("table" -> "db.tbl"))
+
+ assertEquals("db.tbl", task.name)
+ assertEquals(ActionStatus.Success, task.status)
+ assertEquals(Some("compact finished"), task.message)
+ assertEquals("db.tbl", task.metrics("table"))
+ }
+
+ @Test
+ def actionResultUsesSpecDefinedFields(): Unit = {
+ val result =
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks =
+ Seq(ActionTaskResult("db.tbl", ActionStatus.Success, None, Map("table" -> "db.tbl"))),
+ message = Some("all tasks completed"),
+ metrics = Map("table" -> "db.tbl"),
+ costMs = 12.5d)
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals("db.tbl", result.tasks.head.name)
+ assertEquals(Some("all tasks completed"), result.message)
+ assertEquals("db.tbl", result.metrics("table"))
+ assertEquals(12.5d, result.costMs)
+ }
+
+ @Test
+ def producerCoreModelUsesSpecDefinedShape(): Unit = {
+ val config =
+ CommonProducerConfig(
+ format = "paimon",
+ action = "compact",
+ catalogName = "catalog",
+ databaseName = Some("db"),
+ tableName = Some("tbl"),
+ tableNameRegex = None,
+ retryTimes = 3,
+ continueOnTableFailure = true)
+ val table = TargetTable(raw = "db.tbl", database = "db", table = "tbl")
+ val adapter = new DummyAdapter
+ val context = ProducerContext(spark = null, common = config, adapter = adapter)
+ val action = new DummyAction
+
+ assertEquals("paimon", config.format)
+ assertEquals("compact", config.action)
+ assertEquals("db.tbl", table.raw)
+ assertEquals(config, context.common)
+ assertEquals(adapter, context.adapter)
+ assertEquals(Seq(action.name), adapter.actions.map(_.name))
+ assertEquals(RetryMode.FrameworkTableRetry, action.retryMode)
+ }
+
+ final private class DummyConfig extends ActionConfig
+
+ final private class DummyAction extends ProducerAction {
+ override def name: String = "compact"
+
+ override def retryMode: RetryMode = RetryMode.FrameworkTableRetry
+
+ override def options: Seq[ActionOption] = Seq.empty
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = new DummyConfig
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {}
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks =
+ Seq(ActionTaskResult(table.raw, ActionStatus.Success, None, Map("table" -> table.raw))),
+ message = None,
+ metrics = Map("table" -> table.raw),
+ costMs = 0.0d)
+ }
+ }
+
+ final private class DummyAdapter extends LakeFormatAdapter {
+ private val action = new DummyAction
+
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq(action)
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = {
+ Seq(TargetTable("db.tbl", database, "tbl"))
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerAppTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerAppTest.scala
new file mode 100644
index 000000000..c86751e2c
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerAppTest.scala
@@ -0,0 +1,584 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.io.{ByteArrayOutputStream, PrintStream}
+
+import org.apache.spark.sql.SparkSession
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class ProducerAppTest {
+
+ @Test
+ def frameworkTableRetryRetriesThrownAction(): Unit = {
+ val action = new CountingAction(RetryMode.FrameworkTableRetry, failTimes = 2)
+ val result =
+ ProducerApp.executeOneTableWithRetry(
+ context = fakeContext(action),
+ action = action,
+ actionConfig = NoopConfig,
+ table = TargetTable("db.t1", "db", "t1"))
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals(3, action.calls)
+ }
+
+ @Test
+ def actionManagedRetryDoesNotRetryInFramework(): Unit = {
+ val action = new CountingAction(RetryMode.ActionManagedRetry, failTimes = 1)
+
+ assertThrows(
+ classOf[RuntimeException],
+ () =>
+ ProducerApp.executeOneTableWithRetry(
+ fakeContext(action),
+ action,
+ NoopConfig,
+ TargetTable("db.t1", "db", "t1")))
+ assertEquals(1, action.calls)
+ }
+
+ @Test
+ def executeTablesContinuesAfterFailedTableWhenConfigured(): Unit = {
+ val action = new FailFirstTableAction
+ val (output, results) =
+ captureStdout {
+ ProducerApp.executeTables(
+ context = fakeContext(action, continueOnTableFailure = true),
+ action = action,
+ actionConfig = NoopConfig,
+ targets = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")))
+ }
+
+ assertEquals(2, results.size)
+ assertEquals(ActionStatus.Failed, results.head._2.status)
+ assertEquals(ActionStatus.Success, results(1)._2.status)
+ assertEquals(Some("failed table db.t1"), results.head._2.message)
+ assertEquals(Seq("db.t1", "db.t2"), action.seenTables)
+ val jsonLines = tableActionJsonLines(output)
+ assertEquals(2, jsonLines.size)
+ assertTrue(jsonLines.head.contains("\"status\":\"failed\""))
+ assertTrue(jsonLines.head.contains("\"message\":\"failed table db.t1\""))
+ assertTrue(jsonLines(1).contains("\"status\":\"success\""))
+ }
+
+ @Test
+ def executeTablesLogsSuccessfulTableActionJson(): Unit = {
+ val action =
+ new CountingAction(
+ RetryMode.ActionManagedRetry,
+ failTimes = 0,
+ actionName = "expire-snapshots",
+ metrics = Map("deletedSnapshotsTotal" -> "7", "rounds" -> "2"))
+ val (output, results) =
+ captureStdout {
+ ProducerApp.executeTables(
+ context = fakeContext(action),
+ action = action,
+ actionConfig = NoopConfig,
+ targets = Seq(TargetTable("db.t1", "db", "t1")))
+ }
+
+ assertEquals(1, results.size)
+ val json = tableActionJsonLines(output).head
+ assertTrue(json.contains("\"event\":\"lakehouse_producer_table_action\""))
+ assertTrue(json.contains("\"format\":\"paimon\""))
+ assertTrue(json.contains("\"catalog\":\"spark_catalog\""))
+ assertTrue(json.contains("\"database\":\"db\""))
+ assertTrue(json.contains("\"table\":\"t1\""))
+ assertTrue(json.contains("\"target\":\"db.t1\""))
+ assertTrue(json.contains("\"action\":\"expire-snapshots\""))
+ assertTrue(json.contains("\"status\":\"success\""))
+ assertTrue(json.contains("\"costMs\":"))
+ assertTrue(json.contains("\"deletedSnapshotsTotal\":7"))
+ assertTrue(json.contains(
+ "\"metrics\":{\"deletedSnapshotsTotal\":\"7\",\"rounds\":\"2\",\"table\":\"db.t1\"}"))
+ assertTrue(json.contains("\"message\":null"))
+ }
+
+ @Test
+ def executeTablesLogsReturnedFailedJsonBeforeStopping(): Unit = {
+ val action = new ReturnFailedFirstTableAction
+ val (output, thrown) =
+ captureStdout {
+ assertThrows(
+ classOf[RuntimeException],
+ () =>
+ ProducerApp.executeTables(
+ context = fakeContext(action, continueOnTableFailure = false),
+ action = action,
+ actionConfig = NoopConfig,
+ targets =
+ Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2"))))
+ }
+
+ assertEquals(Seq("db.t1"), action.seenTables)
+ assertTrue(thrown.getMessage.contains("failed table db.t1"))
+ val jsonLines = tableActionJsonLines(output)
+ assertEquals(1, jsonLines.size)
+ assertTrue(jsonLines.head.contains("\"status\":\"failed\""))
+ assertTrue(jsonLines.head.contains("\"message\":\"failed table db.t1\""))
+ }
+
+ @Test
+ def tableActionJsonEscapesStringsAndUsesNullSnapshotCountWhenMissing(): Unit = {
+ val table = TargetTable("db.t\"1", "d\\b", "t\n1")
+ val json =
+ TableActionJsonLog.render(
+ common =
+ fakeContext(new CountingAction(RetryMode.ActionManagedRetry, failTimes = 0))
+ .common
+ .copy(
+ format = "pai\"mon",
+ action = "expire\\snapshots",
+ catalogName = "spark\ncatalog"),
+ table = table,
+ result =
+ ActionResult(
+ status = ActionStatus.Failed,
+ tasks = Seq.empty,
+ message = Some("bad \"message\"\\line\nnext"),
+ metrics = Map("key\"1" -> "value\\1\nnext"),
+ costMs = 0.0d),
+ measuredCostMs = 12.5d)
+
+ assertTrue(json.contains("\"format\":\"pai\\\"mon\""))
+ assertTrue(json.contains("\"catalog\":\"spark\\ncatalog\""))
+ assertTrue(json.contains("\"database\":\"d\\\\b\""))
+ assertTrue(json.contains("\"table\":\"t\\n1\""))
+ assertTrue(json.contains("\"action\":\"expire\\\\snapshots\""))
+ assertTrue(json.contains("\"deletedSnapshotsTotal\":null"))
+ assertTrue(json.contains("\"metrics\":{\"key\\\"1\":\"value\\\\1\\nnext\"}"))
+ assertTrue(json.contains("\"message\":\"bad \\\"message\\\"\\\\line\\nnext\""))
+ }
+
+ @Test
+ def tableActionJsonUsesNullSnapshotCountForNonNumericMetric(): Unit = {
+ val json =
+ TableActionJsonLog.render(
+ common =
+ fakeContext(new CountingAction(RetryMode.ActionManagedRetry, failTimes = 0)).common,
+ table = TargetTable("db.t1", "db", "t1"),
+ result =
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks = Seq.empty,
+ message = None,
+ metrics = Map("deletedSnapshotsTotal" -> "unknown"),
+ costMs = 0.0d),
+ measuredCostMs = 1.0d)
+
+ assertTrue(json.contains("\"deletedSnapshotsTotal\":null"))
+ assertTrue(json.contains("\"metrics\":{\"deletedSnapshotsTotal\":\"unknown\"}"))
+ }
+
+ @Test
+ def tableActionJsonIgnoresNullMetricKeysAndRendersNullMetricValues(): Unit = {
+ val metrics =
+ Map(
+ null.asInstanceOf[String] -> "ignored",
+ "deletedSnapshotsTotal" -> null.asInstanceOf[String],
+ "empty" -> null.asInstanceOf[String])
+ val json =
+ TableActionJsonLog.render(
+ common =
+ fakeContext(new CountingAction(RetryMode.ActionManagedRetry, failTimes = 0)).common,
+ table = TargetTable("db.t1", "db", "t1"),
+ result =
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks = Seq.empty,
+ message = None,
+ metrics = metrics,
+ costMs = 0.0d),
+ measuredCostMs = 1.0d)
+
+ assertTrue(json.contains("\"deletedSnapshotsTotal\":null"))
+ assertTrue(json.contains("\"metrics\":{\"deletedSnapshotsTotal\":null,\"empty\":null}"))
+ assertFalse(json.contains("ignored"))
+ }
+
+ @Test
+ def executeTablesContinuesAfterReturnedFailedTableWhenConfigured(): Unit = {
+ val action = new ReturnFailedFirstTableAction
+ val (output, results) =
+ captureStdout {
+ ProducerApp.executeTables(
+ context = fakeContext(action, continueOnTableFailure = true),
+ action = action,
+ actionConfig = NoopConfig,
+ targets = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2")))
+ }
+
+ assertEquals(2, results.size)
+ assertEquals(ActionStatus.Failed, results.head._2.status)
+ assertEquals(ActionStatus.Success, results(1)._2.status)
+ assertEquals(Seq("db.t1", "db.t2"), action.seenTables)
+ val jsonLines = tableActionJsonLines(output)
+ assertEquals(2, jsonLines.size)
+ assertTrue(jsonLines.head.contains("\"status\":\"failed\""))
+ assertTrue(jsonLines(1).contains("\"status\":\"success\""))
+ }
+
+ @Test
+ def executeTablesStopsAfterFailedTableWhenConfigured(): Unit = {
+ val action = new FailFirstTableAction
+ val (output, thrown) =
+ captureStdout {
+ assertThrows(
+ classOf[RuntimeException],
+ () =>
+ ProducerApp.executeTables(
+ context = fakeContext(action, continueOnTableFailure = false),
+ action = action,
+ actionConfig = NoopConfig,
+ targets =
+ Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2"))))
+ }
+
+ assertEquals(Seq("db.t1"), action.seenTables)
+ assertTrue(thrown.getMessage.contains("failed table db.t1"))
+ val jsonLines = tableActionJsonLines(output)
+ assertEquals(1, jsonLines.size)
+ assertTrue(jsonLines.head.contains("\"status\":\"failed\""))
+ assertTrue(jsonLines.head.contains("\"message\":\"failed table db.t1\""))
+ }
+
+ @Test
+ def executeTablesStopsAfterReturnedFailedTableWhenConfigured(): Unit = {
+ val action = new ReturnFailedFirstTableAction
+ val (output, _) =
+ captureStdout {
+ assertThrows(
+ classOf[RuntimeException],
+ () =>
+ ProducerApp.executeTables(
+ context = fakeContext(action, continueOnTableFailure = false),
+ action = action,
+ actionConfig = NoopConfig,
+ targets =
+ Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2"))))
+ }
+
+ assertEquals(Seq("db.t1"), action.seenTables)
+ assertEquals(1, tableActionJsonLines(output).size)
+ }
+
+ @Test
+ def executeTablesFailsFastWhenFrameworkRetryTimesIsInvalid(): Unit = {
+ val action = new CountingAction(RetryMode.FrameworkTableRetry, failTimes = 0)
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ ProducerApp.executeTables(
+ context = fakeContext(action, retryTimes = 0, continueOnTableFailure = true),
+ action = action,
+ actionConfig = NoopConfig,
+ targets = Seq(TargetTable("db.t1", "db", "t1"), TargetTable("db.t2", "db", "t2"))))
+
+ assertTrue(exception.getMessage.contains("retryTimes"))
+ assertEquals(0, action.calls)
+ }
+
+ @Test
+ def resolveAdapterMatchesCaseInsensitiveFormat(): Unit = {
+ val paimon = new FakeAdapter("paimon", Seq.empty)
+ val iceberg = new FakeAdapter("iceberg", Seq.empty)
+
+ assertEquals(paimon, ProducerApp.resolveAdapter("PAIMON", Seq(iceberg, paimon)))
+ }
+
+ @Test
+ def resolveAdapterRejectsMissingFormat(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerApp.resolveAdapter("hudi", Seq(new FakeAdapter("paimon", Seq.empty))))
+ }
+
+ @Test
+ def resolveAdapterRejectsDuplicateNormalizedFormats(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ ProducerApp.resolveAdapter(
+ "paimon",
+ Seq(new FakeAdapter("paimon", Seq.empty), new FakeAdapter(" PAIMON ", Seq.empty))))
+
+ assertTrue(exception.getMessage.contains("paimon"))
+ }
+
+ @Test
+ def resolveActionMatchesCaseInsensitiveName(): Unit = {
+ val compact =
+ new CountingAction(RetryMode.FrameworkTableRetry, failTimes = 0, actionName = "compact")
+ val expire =
+ new CountingAction(RetryMode.FrameworkTableRetry, failTimes = 0, actionName = "expire")
+ val adapter = new FakeAdapter("paimon", Seq(expire, compact))
+
+ assertEquals(compact, ProducerApp.resolveAction(adapter, "COMPACT"))
+ }
+
+ @Test
+ def resolveActionRejectsMissingName(): Unit = {
+ val adapter =
+ new FakeAdapter(
+ "paimon",
+ Seq(new CountingAction(RetryMode.FrameworkTableRetry, failTimes = 0)))
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerApp.resolveAction(adapter, "missing"))
+ }
+
+ @Test
+ def resolveActionRejectsDuplicateNormalizedNames(): Unit = {
+ val adapter =
+ new FakeAdapter(
+ "paimon",
+ Seq(
+ new CountingAction(
+ RetryMode.FrameworkTableRetry,
+ failTimes = 0,
+ actionName = "compact"),
+ new CountingAction(
+ RetryMode.FrameworkTableRetry,
+ failTimes = 0,
+ actionName = " COMPACT ")))
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerApp.resolveAction(adapter, "compact"))
+
+ assertTrue(exception.getMessage.contains("compact"))
+ }
+
+ @Test
+ def producerSummaryCountsTablesAndTasksByStatus(): Unit = {
+ val successTable = TargetTable("db.success", "db", "success")
+ val failedTable = TargetTable("db.failed", "db", "failed")
+ val skippedTable = TargetTable("db.skipped", "db", "skipped")
+ val results =
+ Seq(
+ successTable ->
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks =
+ Seq(
+ ActionTaskResult("success-1", ActionStatus.Success, None, Map.empty),
+ ActionTaskResult("success-2", ActionStatus.Success, None, Map.empty)),
+ message = None,
+ metrics = Map.empty,
+ costMs = 1.0d),
+ failedTable ->
+ ActionResult(
+ status = ActionStatus.Failed,
+ tasks =
+ Seq(
+ ActionTaskResult("failed-1", ActionStatus.Failed, None, Map.empty),
+ ActionTaskResult("failed-skipped", ActionStatus.Skipped, None, Map.empty)),
+ message = Some("failed"),
+ metrics = Map.empty,
+ costMs = 2.0d),
+ skippedTable ->
+ ActionResult(
+ status = ActionStatus.Skipped,
+ tasks = Seq.empty,
+ message = Some("skipped"),
+ metrics = Map.empty,
+ costMs = 0.0d))
+
+ val summary = ProducerSummary.fromResults(results)
+
+ assertEquals(3, summary.tableCount)
+ assertEquals(Seq(successTable), summary.successTables)
+ assertEquals(Seq(failedTable), summary.failedTables)
+ assertEquals(Seq(skippedTable), summary.skippedTables)
+ assertEquals(Seq(FailedTableSummary(failedTable, "failed")), summary.failedTableSummaries)
+ assertEquals(2, summary.taskSuccessCount)
+ assertEquals(1, summary.taskFailedCount)
+ }
+
+ @Test
+ def producerSummaryUsesFailedTaskMessageWhenResultMessageIsMissing(): Unit = {
+ val failedTable = TargetTable("db.failed", "db", "failed")
+ val summary =
+ ProducerSummary.fromResults(
+ Seq(
+ failedTable ->
+ ActionResult(
+ status = ActionStatus.Failed,
+ tasks =
+ Seq(
+ ActionTaskResult(
+ "round-1",
+ ActionStatus.Failed,
+ Some("task failed"),
+ Map.empty)),
+ message = None,
+ metrics = Map.empty,
+ costMs = 2.0d)))
+
+ assertEquals(Seq(FailedTableSummary(failedTable, "task failed")), summary.failedTableSummaries)
+ }
+
+ private def captureStdout[T](block: => T): (String, T) = {
+ val output = new ByteArrayOutputStream()
+ val printStream = new PrintStream(output, true, "UTF-8")
+ val result = Console.withOut(printStream) {
+ block
+ }
+ printStream.flush()
+ output.toString("UTF-8") -> result
+ }
+
+ private def tableActionJsonLines(output: String): Seq[String] = {
+ output.linesIterator.filter(_.contains("\"event\":\"lakehouse_producer_table_action\"")).toSeq
+ }
+
+ private def fakeContext(
+ action: ProducerAction,
+ retryTimes: Int = 3,
+ continueOnTableFailure: Boolean = true): ProducerContext = {
+ ProducerContext(
+ spark = null,
+ common =
+ CommonProducerConfig(
+ format = "paimon",
+ action = action.name,
+ catalogName = "spark_catalog",
+ databaseName = Some("db"),
+ tableName = None,
+ tableNameRegex = None,
+ retryTimes = retryTimes,
+ continueOnTableFailure = continueOnTableFailure),
+ adapter = new FakeAdapter("paimon", Seq(action)))
+ }
+
+ private object NoopConfig extends ActionConfig
+
+ final private class CountingAction(
+ mode: RetryMode,
+ failTimes: Int,
+ actionName: String = "compact",
+ metrics: Map[String, String] = Map.empty)
+ extends ProducerAction {
+ var calls: Int = 0
+
+ override def name: String = actionName
+
+ override def retryMode: RetryMode = mode
+
+ override def options: Seq[ActionOption] = Seq.empty
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = NoopConfig
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {}
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ calls += 1
+ if (calls <= failTimes) {
+ throw new RuntimeException(s"failed attempt $calls")
+ }
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks = Seq(ActionTaskResult(table.raw, ActionStatus.Success, None, Map.empty)),
+ message = None,
+ metrics = metrics + ("table" -> table.raw),
+ costMs = calls.toDouble)
+ }
+ }
+
+ final private class FailFirstTableAction extends ProducerAction {
+ var seenTables: Seq[String] = Seq.empty
+
+ override def name: String = "compact"
+
+ override def retryMode: RetryMode = RetryMode.ActionManagedRetry
+
+ override def options: Seq[ActionOption] = Seq.empty
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = NoopConfig
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {}
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ seenTables = seenTables :+ table.raw
+ if (seenTables.size == 1) {
+ throw new RuntimeException(s"failed table ${table.raw}")
+ }
+ ActionResult(
+ status = ActionStatus.Success,
+ tasks = Seq(ActionTaskResult(table.raw, ActionStatus.Success, None, Map.empty)),
+ message = None,
+ metrics = Map("table" -> table.raw),
+ costMs = 0.0d)
+ }
+ }
+
+ final private class ReturnFailedFirstTableAction extends ProducerAction {
+ var seenTables: Seq[String] = Seq.empty
+
+ override def name: String = "compact"
+
+ override def retryMode: RetryMode = RetryMode.ActionManagedRetry
+
+ override def options: Seq[ActionOption] = Seq.empty
+
+ override def parse(commandLine: ProducerParsedOptions): ActionConfig = NoopConfig
+
+ override def validate(common: CommonProducerConfig, actionConfig: ActionConfig): Unit = {}
+
+ override def execute(
+ context: ProducerContext,
+ table: TargetTable,
+ actionConfig: ActionConfig): ActionResult = {
+ seenTables = seenTables :+ table.raw
+ val status = if (seenTables.size == 1) ActionStatus.Failed else ActionStatus.Success
+ ActionResult(
+ status = status,
+ tasks = Seq(ActionTaskResult(table.raw, status, None, Map.empty)),
+ message = if (status == ActionStatus.Failed) Some(s"failed table ${table.raw}") else None,
+ metrics = Map("table" -> table.raw),
+ costMs = 0.0d)
+ }
+ }
+
+ final private class FakeAdapter(formatName: String, adapterActions: Seq[ProducerAction])
+ extends LakeFormatAdapter {
+ override def format: String = formatName
+
+ override def actions: Seq[ProducerAction] = adapterActions
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = Seq.empty
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerCliParserTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerCliParserTest.scala
new file mode 100644
index 000000000..a518c37d2
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/ProducerCliParserTest.scala
@@ -0,0 +1,246 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import org.junit.jupiter.api.Assertions.{assertEquals, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+class ProducerCliParserTest {
+
+ @Test
+ def peekReadsFormatAndActionWithoutBeingDisturbedByActionOptions(): Unit = {
+ val peek =
+ ProducerCliParser.peek(
+ Array(
+ "--startBucket",
+ "10",
+ "--format",
+ "paimon",
+ "--endBucket",
+ "20",
+ "--action",
+ "compact"))
+
+ assertEquals(Some("paimon"), peek.format)
+ assertEquals(Some("compact"), peek.action)
+ assertEquals(false, peek.help)
+ }
+
+ @Test
+ def peekTrimsFormatAndActionValues(): Unit = {
+ val peek =
+ ProducerCliParser.peek(Array("--format", " paimon ", "--action", " compact "))
+
+ assertEquals(Some("paimon"), peek.format)
+ assertEquals(Some("compact"), peek.action)
+ }
+
+ @Test
+ def peekAcceptsLongOptionsWithEquals(): Unit = {
+ val peek = ProducerCliParser.peek(Array("--format=paimon", "--action=compact"))
+
+ assertEquals(Some("paimon"), peek.format)
+ assertEquals(Some("compact"), peek.action)
+ }
+
+ @Test
+ def peekFiltersBlankFormatAndActionValues(): Unit = {
+ val peek = ProducerCliParser.peek(Array("--format", " ", "--action", ""))
+
+ assertEquals(None, peek.format)
+ assertEquals(None, peek.action)
+ }
+
+ @Test
+ def parseCommonUsesDefaults(): Unit = {
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array("--format", "paimon", "--action", "compact"))
+
+ val config = ProducerCliParser.parseCommon(commandLine)
+
+ assertEquals("paimon", config.format)
+ assertEquals("compact", config.action)
+ assertEquals("spark_catalog", config.catalogName)
+ assertEquals(None, config.databaseName)
+ assertEquals(None, config.tableName)
+ assertEquals(None, config.tableNameRegex)
+ assertEquals(3, config.retryTimes)
+ assertEquals(true, config.continueOnTableFailure)
+ }
+
+ @Test
+ def parseCommonTrimsStringValues(): Unit = {
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array(
+ "--format",
+ " paimon ",
+ "--action",
+ " compact ",
+ "--catalogName",
+ " paimon ",
+ "--databaseName",
+ " db ",
+ "--tableName",
+ " orders ",
+ "--tableNameRegex",
+ " order "))
+
+ val config = ProducerCliParser.parseCommon(commandLine)
+
+ assertEquals("paimon", config.format)
+ assertEquals("compact", config.action)
+ assertEquals("paimon", config.catalogName)
+ assertEquals(Some("db"), config.databaseName)
+ assertEquals(Some("orders"), config.tableName)
+ assertEquals(Some("order"), config.tableNameRegex)
+ }
+
+ @Test
+ def parseCommonAcceptsTrimmedBooleanValue(): Unit = {
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array(
+ "--format",
+ "paimon",
+ "--action",
+ "compact",
+ "--continueOnTableFailure",
+ " TRUE "))
+
+ val config = ProducerCliParser.parseCommon(commandLine)
+
+ assertEquals(true, config.continueOnTableFailure)
+ }
+
+ @Test
+ def parseCommonRejectsZeroRetryTimes(): Unit = {
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array("--format", "paimon", "--action", "compact", "--retryTimes", "0"))
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerCliParser.parseCommon(commandLine))
+ }
+
+ @Test
+ def parseCommonRejectsIllegalBooleanValue(): Unit = {
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array(
+ "--format",
+ "paimon",
+ "--action",
+ "compact",
+ "--continueOnTableFailure",
+ "maybe"))
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerCliParser.parseCommon(commandLine))
+ }
+
+ @Test
+ def parseMergedAcceptsRegisteredActionOption(): Unit = {
+ val actionOptions = Seq(ActionOption("startBucket", description = "start bucket"))
+
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ actionOptions,
+ Array("--format", "paimon", "--action", "compact", "--startBucket", "10"))
+
+ assertEquals("10", commandLine.getOptionValue("startBucket"))
+ }
+
+ @Test
+ def parseMergedAcceptsLongOptionsWithEquals(): Unit = {
+ val actionOptions = Seq(ActionOption("startBucket", description = "start bucket"))
+
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ actionOptions,
+ Array("--format=paimon", "--action=compact", "--startBucket=-1"))
+
+ assertEquals("paimon", commandLine.getOptionValue("format"))
+ assertEquals("compact", commandLine.getOptionValue("action"))
+ assertEquals("-1", commandLine.getOptionValue("startBucket"))
+ }
+
+ @Test
+ def parseMergedAcceptsValueStartingWithDashWhenItIsNotARegisteredOption(): Unit = {
+ val actionOptions = Seq(ActionOption("startBucket", description = "start bucket"))
+
+ val commandLine =
+ ProducerCliParser.parseMerged(
+ actionOptions,
+ Array("--format", "paimon", "--action", "compact", "--startBucket", "-1"))
+
+ assertEquals("-1", commandLine.getOptionValue("startBucket"))
+ }
+
+ @Test
+ def mergeOptionsRejectsActionOptionWithCommonLongOpt(): Unit = {
+ val actionOptions = Seq(ActionOption("format", description = "bad"))
+
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerCliParser.mergeOptions(actionOptions))
+
+ assertTrue(exception.getMessage.contains("format"))
+ }
+
+ @Test
+ def mergeOptionsRejectsActionOptionWithCommonShortOpt(): Unit = {
+ val actionOptions = Seq(ActionOption("customHelp", hasValue = false, shortName = Some("h")))
+
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerCliParser.mergeOptions(actionOptions))
+
+ assertTrue(exception.getMessage.contains("h"))
+ }
+
+ @Test
+ def parseMergedRejectsUnregisteredOption(): Unit = {
+ assertThrows(
+ classOf[ProducerCliParseException],
+ () =>
+ ProducerCliParser.parseMerged(
+ Seq.empty,
+ Array("--format", "paimon", "--action", "compact", "--badOption", "value")))
+ }
+
+ @Test
+ def peekRecognizesHelp(): Unit = {
+ val peek = ProducerCliParser.peek(Array("--help"))
+
+ assertEquals(None, peek.format)
+ assertEquals(None, peek.action)
+ assertEquals(true, peek.help)
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableFilterTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableFilterTest.scala
new file mode 100644
index 000000000..051c8e6b7
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableFilterTest.scala
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import org.junit.jupiter.api.Assertions.{assertEquals, assertThrows}
+import org.junit.jupiter.api.Test
+
+class TableFilterTest {
+
+ @Test
+ def filterUsesFindSemanticsOnTableNameOnly(): Unit = {
+ val tables =
+ Seq(
+ TargetTable("warehouse.orders", "warehouse", "orders"),
+ TargetTable("warehouse.customers", "warehouse", "customers"),
+ TargetTable("matched_by_raw.events", "matched_by_raw", "events"))
+
+ assertEquals(
+ Seq(TargetTable("warehouse.orders", "warehouse", "orders")),
+ TableFilter.filterByRegex(tables, Some("der")))
+ assertEquals(Seq.empty, TableFilter.filterByRegex(tables, Some("matched_by_raw")))
+ }
+
+ @Test
+ def filterRejectsInvalidRegex(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => TableFilter.compileTableNameRegex(""))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => TableFilter.compileTableNameRegex(" "))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => TableFilter.filterByRegex(Seq.empty, Some("[")))
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableResolverTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableResolverTest.scala
new file mode 100644
index 000000000..379979099
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/core/TableResolverTest.scala
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.core
+
+import java.util.regex.PatternSyntaxException
+
+import org.apache.spark.sql.SparkSession
+import org.junit.jupiter.api.Assertions.{assertEquals, assertThrows}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.spi.{LakeFormatAdapter, ProducerAction}
+
+class TableResolverTest {
+
+ @Test
+ def resolveDatabaseOnlyListsTables(): Unit = {
+ val adapter = new FakeAdapter(Seq(TargetTable("db.orders", "db", "orders")))
+
+ val targets = TableResolver.resolveTargets(null, adapter, common(databaseName = Some("db")))
+
+ assertEquals(Seq(TargetTable("db.orders", "db", "orders")), targets)
+ assertEquals(Seq("db"), adapter.listedDatabases)
+ }
+
+ @Test
+ def resolveDatabaseWithSimpleTableReturnsSingleTarget(): Unit = {
+ val adapter = new FakeAdapter(Seq.empty)
+
+ val targets =
+ TableResolver.resolveTargets(
+ null,
+ adapter,
+ common(databaseName = Some("db"), tableName = Some("orders")))
+
+ assertEquals(Seq(TargetTable("db.orders", "db", "orders")), targets)
+ assertEquals(Seq.empty, adapter.listedDatabases)
+ }
+
+ @Test
+ def resolveQualifiedTableOnlyReturnsSingleTarget(): Unit = {
+ val adapter = new FakeAdapter(Seq.empty)
+
+ val targets =
+ TableResolver.resolveTargets(null, adapter, common(tableName = Some("db.orders")))
+
+ assertEquals(Seq(TargetTable("db.orders", "db", "orders")), targets)
+ assertEquals(Seq.empty, adapter.listedDatabases)
+ }
+
+ @Test
+ def resolveRejectsSimpleTableOnly(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ TableResolver.resolveTargets(
+ null,
+ new FakeAdapter(Seq.empty),
+ common(tableName = Some("orders"))))
+ }
+
+ @Test
+ def resolveRejectsDatabaseWithQualifiedTable(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ TableResolver.resolveTargets(
+ null,
+ new FakeAdapter(Seq.empty),
+ common(databaseName = Some("db"), tableName = Some("db.orders"))))
+ }
+
+ @Test
+ def resolveRejectsRegexWithTableName(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ TableResolver.resolveTargets(
+ null,
+ new FakeAdapter(Seq.empty),
+ common(
+ databaseName = Some("db"),
+ tableName = Some("orders"),
+ tableNameRegex = Some("ord"))))
+ }
+
+ @Test
+ def resolveFiltersDatabaseOnlyWithRegex(): Unit = {
+ val adapter =
+ new FakeAdapter(
+ Seq(
+ TargetTable("db.orders", "db", "orders"),
+ TargetTable("db.customers", "db", "customers"),
+ TargetTable("db.order_items", "db", "order_items")))
+
+ val targets =
+ TableResolver.resolveTargets(
+ null,
+ adapter,
+ common(databaseName = Some("db"), tableNameRegex = Some("order")))
+
+ assertEquals(
+ Seq(
+ TargetTable("db.orders", "db", "orders"),
+ TargetTable("db.order_items", "db", "order_items")),
+ targets)
+ assertEquals(Seq("db"), adapter.listedDatabases)
+ }
+
+ @Test
+ def resolveRejectsEmptyRegex(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ TableResolver.resolveTargets(
+ null,
+ new FakeAdapter(Seq.empty),
+ common(databaseName = Some("db"), tableNameRegex = Some(" "))))
+ }
+
+ @Test
+ def resolveRejectsInvalidRegexBeforeListingTables(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ TableResolver.resolveTargets(
+ null,
+ new FailingListAdapter,
+ common(databaseName = Some("db"), tableNameRegex = Some("["))))
+
+ assertEquals(classOf[PatternSyntaxException], exception.getCause.getClass)
+ }
+
+ @Test
+ def resolveRejectsMissingDatabaseAndTable(): Unit = {
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => TableResolver.resolveTargets(null, new FakeAdapter(Seq.empty), common()))
+ }
+
+ private def common(
+ databaseName: Option[String] = None,
+ tableName: Option[String] = None,
+ tableNameRegex: Option[String] = None): CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "compact",
+ catalogName = "spark_catalog",
+ databaseName = databaseName,
+ tableName = tableName,
+ tableNameRegex = tableNameRegex,
+ retryTimes = 3,
+ continueOnTableFailure = true)
+ }
+
+ final private class FakeAdapter(tables: Seq[TargetTable]) extends LakeFormatAdapter {
+ var listedDatabases: Seq[String] = Seq.empty
+
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = {
+ listedDatabases = listedDatabases :+ database
+ tables
+ }
+ }
+
+ final private class FailingListAdapter extends LakeFormatAdapter {
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = {
+ throw new AssertionError("listTables must not be called before regex validation")
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactActionTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactActionTest.scala
new file mode 100644
index 000000000..0424c6e59
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactActionTest.scala
@@ -0,0 +1,395 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{ActionStatus, CommonProducerConfig, ProducerCliParser, ProducerContext, RetryMode, TargetTable}
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class PaimonCompactActionTest {
+
+ @Test
+ def parseUsesScriptDefaults(): Unit = {
+ val action = new PaimonCompactAction()
+ val config = parse(action)
+
+ assertEquals("compact", action.name)
+ assertEquals(RetryMode.ActionManagedRetry, action.retryMode)
+ assertEquals(0, config.startBucket)
+ assertEquals(20, config.step)
+ assertEquals("full", config.compactStrategy)
+ assertEquals("target-file-size=256m", config.procedureOptions)
+ assertEquals("", config.version)
+ assertEquals("1d", config.partitionIdleTime)
+ }
+
+ @Test
+ def validateRejectsInvalidActionArguments(): Unit = {
+ val action = new PaimonCompactAction()
+ val common = commonConfig()
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonCompactConfig(-1, 20, "full", "", "", "1d")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonCompactConfig(0, 0, "full", "", "", "1d")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonCompactConfig(0, 20, " ", "", "", "1d")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonCompactConfig(0, 20, "full", "", "1.0", "1d")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonCompactConfig(0, 20, "full", "", "0.9", " ")))
+ }
+
+ @Test
+ def validateRejectsWrongActionConfigType(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => new PaimonCompactAction().validate(commonConfig(), WrongConfig))
+
+ assertTrue(exception.getMessage.contains("compact"))
+ assertTrue(exception.getMessage.contains("PaimonCompactConfig"))
+ }
+
+ @Test
+ def executeRejectsNonPositiveRetryTimesBeforeRunningSql(): Unit = {
+ val runner = new FakeRunner()
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => new PaimonCompactAction(runner).execute(context(retryTimes = 0), table, defaultConfig))
+ assertTrue(runner.sqls.isEmpty)
+ }
+
+ @Test
+ def parseTrimsActionArguments(): Unit = {
+ val action = new PaimonCompactAction()
+ val config =
+ parse(
+ action,
+ Array(
+ "--startBucket",
+ " 1 ",
+ "--step",
+ " 2 ",
+ "--compactStrategy",
+ " full ",
+ "--procedureOptions",
+ " target-file-size=256m ",
+ "--version",
+ " 0.9 ",
+ "--partitionIdleTime",
+ " 1d "))
+
+ assertEquals(1, config.startBucket)
+ assertEquals(2, config.step)
+ assertEquals("full", config.compactStrategy)
+ assertEquals("target-file-size=256m", config.procedureOptions)
+ assertEquals("0.9", config.version)
+ assertEquals("1d", config.partitionIdleTime)
+ }
+
+ @Test
+ def buildCompactSqlUsesTrimmedActionArguments(): Unit = {
+ val action = new PaimonCompactAction()
+ val config =
+ parse(
+ action,
+ Array(
+ "--startBucket",
+ " 1 ",
+ "--step",
+ " 2 ",
+ "--compactStrategy",
+ " full ",
+ "--procedureOptions",
+ " target-file-size=256m ",
+ "--version",
+ " 0.9 ",
+ "--partitionIdleTime",
+ " 1d "))
+
+ val bucketSql =
+ PaimonCompactAction.buildCompactSql(
+ catalogName = "paimon",
+ config = config.copy(version = ""),
+ table = table,
+ bucketRange = "1-2")
+ val paimon09Sql =
+ PaimonCompactAction.buildCompactSql(
+ catalogName = "paimon",
+ config = config,
+ table = table,
+ bucketRange = "1-2")
+
+ assertTrue(bucketSql.contains("compact_strategy => 'full'"))
+ assertTrue(bucketSql.contains("options => 'target-file-size=256m'"))
+ assertFalse(bucketSql.contains("compact_strategy => ' full '"))
+ assertFalse(bucketSql.contains("options => ' target-file-size=256m '"))
+ assertTrue(paimon09Sql.contains("partition_idle_time => '1d'"))
+ assertFalse(paimon09Sql.contains("partition_idle_time => ' 1d '"))
+ }
+
+ @Test
+ def buildBucketRangesUsesClosedIntervals(): Unit = {
+ assertEquals(
+ Seq("0-19", "20-39", "40-40"),
+ PaimonCompactAction.buildBucketRanges(startBucket = 0, step = 20, bucketNum = 41))
+ }
+
+ @Test
+ def buildCompactSqlIncludesCatalogBucketsStrategyAndOptions(): Unit = {
+ val config =
+ PaimonCompactConfig(
+ startBucket = 0,
+ step = 20,
+ compactStrategy = "full's",
+ procedureOptions = "target-file-size=256m,path='x'",
+ version = "",
+ partitionIdleTime = "1d")
+
+ val sql =
+ PaimonCompactAction.buildCompactSql(
+ catalogName = "pa`imon",
+ config = config,
+ table = table,
+ bucketRange = "0-19")
+
+ assertTrue(sql.contains("CALL `pa``imon`.sys.compact("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("buckets => '0-19'"))
+ assertTrue(sql.contains("compact_strategy => 'full''s'"))
+ assertTrue(sql.contains("options => 'target-file-size=256m,path=''x'''"))
+ }
+
+ @Test
+ def buildCompactSqlOmitsOptionsWhenProcedureOptionsIsEmpty(): Unit = {
+ val sql =
+ PaimonCompactAction.buildCompactSql(
+ catalogName = "paimon",
+ config = PaimonCompactConfig(0, 20, "full", "", "", "1d"),
+ table = table,
+ bucketRange = "0-19")
+
+ assertTrue(sql.contains("compact_strategy => 'full'"))
+ assertFalse(sql.contains("options =>"))
+ }
+
+ @Test
+ def buildCompactSqlUsesPaimon09PartitionIdleTimeOnly(): Unit = {
+ val sql =
+ PaimonCompactAction.buildCompactSql(
+ catalogName = "paimon",
+ config = PaimonCompactConfig(0, 20, "full", "target-file-size=256m", "0.9", "1d"),
+ table = table,
+ bucketRange = "0-19")
+
+ assertTrue(sql.contains("CALL sys.compact("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("partition_idle_time => '1d'"))
+ assertFalse(sql.contains("buckets =>"))
+ assertFalse(sql.contains("compact_strategy =>"))
+ assertFalse(sql.contains("options =>"))
+ }
+
+ @Test
+ def buildNonBucketCompactSqlUsesSimpleSysCompact(): Unit = {
+ val sql = PaimonCompactAction.buildNonBucketCompactSql(table)
+
+ assertTrue(sql.contains("CALL sys.compact("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertFalse(sql.contains("partition_idle_time =>"))
+ assertFalse(sql.contains("buckets =>"))
+ assertFalse(sql.contains("compact_strategy =>"))
+ assertFalse(sql.contains("options =>"))
+ }
+
+ @Test
+ def executeUsesNonBucketSimpleCompactWhenBucketReadFailsOrMissing(): Unit = {
+ val failingBucketRunner = new FakeRunner(bucketFailure = true)
+ val missingBucketRunner = new FakeRunner(bucketRows = Seq.empty)
+
+ new PaimonCompactAction(failingBucketRunner).execute(context(), table, defaultConfig)
+ new PaimonCompactAction(missingBucketRunner).execute(context(), table, defaultConfig)
+
+ assertEquals(Seq("CALL sys.compact(\ntable => 'db.t1'\n)"), compactSqls(failingBucketRunner))
+ assertEquals(Seq("CALL sys.compact(\ntable => 'db.t1'\n)"), compactSqls(missingBucketRunner))
+ }
+
+ @Test
+ def executeUsesNonBucketSimpleCompactWhenBucketValueIsInvalid(): Unit = {
+ val nonIntegerBucketRunner = new FakeRunner(bucketRows = Seq(Row("abc")))
+ val nonPositiveBucketRunner = new FakeRunner(bucketRows = Seq(Row("0")))
+
+ new PaimonCompactAction(nonIntegerBucketRunner).execute(context(), table, defaultConfig)
+ new PaimonCompactAction(nonPositiveBucketRunner).execute(context(), table, defaultConfig)
+
+ assertEquals(Seq("CALL sys.compact(\ntable => 'db.t1'\n)"), compactSqls(nonIntegerBucketRunner))
+ assertEquals(
+ Seq("CALL sys.compact(\ntable => 'db.t1'\n)"),
+ compactSqls(nonPositiveBucketRunner))
+ }
+
+ @Test
+ def executeSkipsWhenStartBucketIsGreaterThanMaxBucketId(): Unit = {
+ val runner = new FakeRunner(bucketRows = Seq(Row("3")))
+ val result =
+ new PaimonCompactAction(runner)
+ .execute(context(), table, defaultConfig.copy(startBucket = 3))
+
+ assertEquals(ActionStatus.Skipped, result.status)
+ assertTrue(compactSqls(runner).isEmpty)
+ }
+
+ @Test
+ def executeRetriesFailedRangeUntilSuccess(): Unit = {
+ val runner = new FakeRunner(bucketRows = Seq(Row("41")), failAttemptsByRange = Map("0-19" -> 1))
+ val result =
+ new PaimonCompactAction(runner).execute(context(retryTimes = 2), table, defaultConfig)
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals(2, compactSqls(runner).count(_.contains("buckets => '0-19'")))
+ assertEquals(ActionStatus.Success, result.tasks.find(_.name == "0-19").get.status)
+ }
+
+ @Test
+ def executeContinuesAfterRangeRetryExhaustedAndReturnsFailedResult(): Unit = {
+ val runner = new FakeRunner(bucketRows = Seq(Row("41")), failAttemptsByRange = Map("0-19" -> 2))
+ val result =
+ new PaimonCompactAction(runner).execute(context(retryTimes = 2), table, defaultConfig)
+
+ assertEquals(ActionStatus.Failed, result.status)
+ assertEquals(ActionStatus.Failed, result.tasks.find(_.name == "0-19").get.status)
+ assertTrue(compactSqls(runner).exists(_.contains("buckets => '20-39'")))
+ assertTrue(compactSqls(runner).exists(_.contains("buckets => '40-40'")))
+ }
+
+ @Test
+ def executePaimon09BucketTableRunsSinglePartitionIdleTimeTask(): Unit = {
+ val runner = new FakeRunner(bucketRows = Seq(Row("41")))
+ val result =
+ new PaimonCompactAction(runner)
+ .execute(context(), table, defaultConfig.copy(version = "0.9", partitionIdleTime = "1d"))
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals(Seq("partition_idle_time=1d"), result.tasks.map(_.name))
+ assertEquals(1, compactSqls(runner).size)
+ assertTrue(compactSqls(runner).head.contains("partition_idle_time => '1d'"))
+ assertFalse(compactSqls(runner).head.contains("buckets =>"))
+ assertFalse(compactSqls(runner).head.contains("compact_strategy =>"))
+ assertFalse(compactSqls(runner).head.contains("options =>"))
+ }
+
+ private def parse(
+ action: PaimonCompactAction,
+ args: Array[String] = Array.empty): PaimonCompactConfig = {
+ val commandLine = ProducerCliParser.parseMerged(action.options, args)
+ action.parse(commandLine).asInstanceOf[PaimonCompactConfig]
+ }
+
+ private def compactSqls(runner: FakeRunner): Seq[String] = {
+ runner.sqls.filterNot(_.contains("$options")).map(normalizeSql)
+ }
+
+ private def normalizeSql(sqlText: String): String = {
+ sqlText.stripMargin.trim.linesIterator.map(_.trim).mkString("\n")
+ }
+
+ private def context(retryTimes: Int = 3): ProducerContext = {
+ ProducerContext(null.asInstanceOf[SparkSession], commonConfig(retryTimes), DummyAdapter)
+ }
+
+ private def commonConfig(retryTimes: Int = 3): CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "compact",
+ catalogName = "paimon",
+ databaseName = Some("db"),
+ tableName = Some("t1"),
+ tableNameRegex = None,
+ retryTimes = retryTimes,
+ continueOnTableFailure = true)
+ }
+
+ private val table = TargetTable("db.t1", "db", "t1")
+
+ private object WrongConfig extends ActionConfig
+
+ private val defaultConfig =
+ PaimonCompactConfig(
+ startBucket = 0,
+ step = 20,
+ compactStrategy = "full",
+ procedureOptions = "target-file-size=256m",
+ version = "",
+ partitionIdleTime = "1d")
+
+ private object DummyAdapter extends LakeFormatAdapter {
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = Seq.empty
+ }
+
+ private class FakeRunner(
+ bucketRows: Seq[Row] = Seq(Row("41")),
+ bucketFailure: Boolean = false,
+ failAttemptsByRange: Map[String, Int] = Map.empty)
+ extends PaimonSqlRunner {
+
+ var sqls: Seq[String] = Seq.empty
+ private var attemptsByRange: Map[String, Int] = Map.empty
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ sqls :+= sqlText
+ if (sqlText.contains("$options")) {
+ if (bucketFailure) {
+ throw new RuntimeException("bucket read failed")
+ }
+ bucketRows
+ } else {
+ val range = bucketRange(sqlText)
+ range.foreach { value =>
+ val nextAttempt = attemptsByRange.getOrElse(value, 0) + 1
+ attemptsByRange += value -> nextAttempt
+ if (nextAttempt <= failAttemptsByRange.getOrElse(value, 0)) {
+ throw new RuntimeException(s"failed range $value")
+ }
+ }
+ Seq.empty
+ }
+ }
+
+ private def bucketRange(sqlText: String): Option[String] = {
+ "'([0-9]+-[0-9]+)'".r.findFirstMatchIn(sqlText).map(_.group(1))
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestActionTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestActionTest.scala
new file mode 100644
index 000000000..7de9516c9
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonCompactManifestActionTest.scala
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.types.{BooleanType, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{ActionStatus, CommonProducerConfig, ProducerCliParser, ProducerContext, RetryMode, TargetTable}
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class PaimonCompactManifestActionTest {
+
+ @Test
+ def parseUsesDefaultsAndTrimsProcedureOptions(): Unit = {
+ val action = new PaimonCompactManifestAction()
+
+ assertEquals("compact-manifest", action.name)
+ assertEquals(RetryMode.FrameworkTableRetry, action.retryMode)
+ assertEquals("", parse(action).procedureOptions)
+ assertEquals(
+ "k=''v''",
+ parse(action, Array("--procedureOptions", " k=''v'' ")).procedureOptions)
+ }
+
+ @Test
+ def buildSqlIncludesEscapedOptionsWhenProcedureOptionsIsNonEmpty(): Unit = {
+ val sql =
+ PaimonCompactManifestAction.buildCompactManifestSql(
+ catalogName = "pa`imon",
+ config = PaimonCompactManifestConfig("k='v'"),
+ table = table)
+
+ assertTrue(sql.contains("CALL `pa``imon`.sys.compact_manifest("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("options => 'k=''v'''"))
+ }
+
+ @Test
+ def buildSqlOmitsOptionsWhenProcedureOptionsIsEmpty(): Unit = {
+ val sql =
+ PaimonCompactManifestAction.buildCompactManifestSql(
+ catalogName = "paimon",
+ config = PaimonCompactManifestConfig(""),
+ table = table)
+
+ assertTrue(sql.contains("CALL `paimon`.sys.compact_manifest("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertFalse(sql.contains("options =>"))
+ }
+
+ @Test
+ def validateRejectsWrongActionConfigType(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => new PaimonCompactManifestAction().validate(commonConfig(), WrongConfig))
+
+ assertTrue(exception.getMessage.contains("compact-manifest"))
+ assertTrue(exception.getMessage.contains("PaimonCompactManifestConfig"))
+ }
+
+ @Test
+ def executeUsesRunnerOnceAndParsesBooleanResultField(): Unit = {
+ val config = PaimonCompactManifestConfig("k='v'")
+ val runner =
+ new FakeRunner(
+ Seq(
+ new GenericRowWithSchema(
+ Array[Any](java.lang.Boolean.TRUE),
+ StructType(Seq(StructField("result", BooleanType))))))
+ val result =
+ new PaimonCompactManifestAction(runner)
+ .execute(context(), table, config)
+
+ assertEquals(1, runner.sqls.size)
+ assertTrue(runner.sqls.head.contains("CALL `paimon`.sys.compact_manifest("))
+ assertTrue(runner.sqls.head.contains("table => 'db.t1'"))
+ assertTrue(runner.sqls.head.contains("options => 'k=''v'''"))
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals("true", result.metrics("result"))
+ assertEquals(Seq(ActionStatus.Success), result.tasks.map(_.status))
+ assertEquals(Seq("compact_manifest"), result.tasks.map(_.name))
+ assertTrue(result.costMs >= 0.0)
+ }
+
+ @Test
+ def executeParsesStringAndFirstColumnBooleanResults(): Unit = {
+ val stringRunner = new FakeRunner(Seq(Row("1")))
+ val falseRunner = new FakeRunner(Seq(Row(false)))
+
+ val stringResult =
+ new PaimonCompactManifestAction(stringRunner)
+ .execute(context(), table, PaimonCompactManifestConfig(""))
+ val falseResult =
+ new PaimonCompactManifestAction(falseRunner)
+ .execute(context(), table, PaimonCompactManifestConfig(""))
+
+ assertEquals("true", stringResult.metrics("result"))
+ assertEquals("false", falseResult.metrics("result"))
+ }
+
+ private def parse(
+ action: PaimonCompactManifestAction,
+ args: Array[String] = Array.empty): PaimonCompactManifestConfig = {
+ val commandLine = ProducerCliParser.parseMerged(action.options, args)
+ action.parse(commandLine).asInstanceOf[PaimonCompactManifestConfig]
+ }
+
+ private def context(): ProducerContext = {
+ ProducerContext(null.asInstanceOf[SparkSession], commonConfig(), DummyAdapter)
+ }
+
+ private def commonConfig(): CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "compact-manifest",
+ catalogName = "paimon",
+ databaseName = Some("db"),
+ tableName = Some("t1"),
+ tableNameRegex = None,
+ retryTimes = 3,
+ continueOnTableFailure = true)
+ }
+
+ private val table = TargetTable("db.t1", "db", "t1")
+
+ private object WrongConfig extends ActionConfig
+
+ private object DummyAdapter extends LakeFormatAdapter {
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = Seq.empty
+ }
+
+ private class FakeRunner(rows: Seq[Row]) extends PaimonSqlRunner {
+ var sqls: Seq[String] = Seq.empty
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ sqls :+= sqlText
+ rows
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsActionTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsActionTest.scala
new file mode 100644
index 000000000..3504d28fd
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonExpireSnapshotsActionTest.scala
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.types.{LongType, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertFalse, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{ActionStatus, CommonProducerConfig, ProducerCliParser, ProducerContext, RetryMode, TargetTable}
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class PaimonExpireSnapshotsActionTest {
+
+ @Test
+ def parseUsesDefaultsAndTrimsArguments(): Unit = {
+ val action = new PaimonExpireSnapshotsAction()
+
+ assertEquals("expire-snapshots", action.name)
+ assertEquals(RetryMode.ActionManagedRetry, action.retryMode)
+ assertEquals(
+ PaimonExpireSnapshotsConfig(
+ retainMax = 20,
+ retainMin = 10,
+ maxDeletes = 550L,
+ procedureOptions = "file-operation.thread-num=32,snapshot.expire.execution-mode=sync"),
+ parse(action))
+ assertEquals(
+ PaimonExpireSnapshotsConfig(2, 1, 9L, "k=v"),
+ parse(
+ action,
+ Array(
+ "--retainMax",
+ " 2 ",
+ "--retainMin",
+ " 1 ",
+ "--maxDeletes",
+ " 9 ",
+ "--procedureOptions",
+ " k=v ")))
+ }
+
+ @Test
+ def buildSqlIncludesCatalogTableRetainLimitsMaxDeletesAndOptions(): Unit = {
+ val sql =
+ PaimonExpireSnapshotsAction.buildExpireSnapshotsSql(
+ catalogName = "pa`imon",
+ config = PaimonExpireSnapshotsConfig(20, 10, 550L, "k='v'"),
+ table = table)
+
+ assertTrue(sql.contains("CALL `pa``imon`.sys.expire_snapshots("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("retain_max => 20"))
+ assertTrue(sql.contains("retain_min => 10"))
+ assertTrue(sql.contains("max_deletes => 550"))
+ assertTrue(sql.contains("options => 'k=''v'''"))
+ }
+
+ @Test
+ def buildSqlOmitsOptionsWhenProcedureOptionsIsEmpty(): Unit = {
+ val sql =
+ PaimonExpireSnapshotsAction.buildExpireSnapshotsSql(
+ catalogName = "paimon",
+ config = PaimonExpireSnapshotsConfig(20, 10, 550L, " "),
+ table = table)
+
+ assertTrue(sql.contains("CALL `paimon`.sys.expire_snapshots("))
+ assertTrue(sql.contains("max_deletes => 550"))
+ assertFalse(sql.contains("options =>"))
+ }
+
+ @Test
+ def validateRejectsInvalidArguments(): Unit = {
+ val action = new PaimonExpireSnapshotsAction()
+ val common = commonConfig()
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonExpireSnapshotsConfig(-1, 10, 550L, "")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonExpireSnapshotsConfig(20, -1, 550L, "")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonExpireSnapshotsConfig(20, 10, -1L, "")))
+ val error =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonExpireSnapshotsConfig(20, 10, 0L, "")))
+ assertEquals("--maxDeletes 必须大于 0", error.getMessage)
+ }
+
+ @Test
+ def validateRejectsWrongActionConfigType(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => new PaimonExpireSnapshotsAction().validate(commonConfig(), WrongConfig))
+
+ assertTrue(exception.getMessage.contains("expire-snapshots"))
+ assertTrue(exception.getMessage.contains("PaimonExpireSnapshotsConfig"))
+ }
+
+ @Test
+ def executeRejectsZeroMaxDeletesBeforeRunningSql(): Unit = {
+ val runner = new FakeRunner(Seq(success(Row(20L))))
+ val error =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ new PaimonExpireSnapshotsAction(runner)
+ .execute(context(), table, defaultConfig.copy(maxDeletes = 0L)))
+
+ assertEquals("--maxDeletes 必须大于 0", error.getMessage)
+ assertTrue(runner.sqls.isEmpty)
+ }
+
+ @Test
+ def executeRejectsNonPositiveRetryTimesBeforeRunningSql(): Unit = {
+ val runner = new FakeRunner(Seq(success(Row(20L))))
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () =>
+ new PaimonExpireSnapshotsAction(runner)
+ .execute(context(retryTimes = 0), table, defaultConfig))
+ assertTrue(runner.sqls.isEmpty)
+ }
+
+ @Test
+ def executeRepeatsRoundsUntilDeletedCountIsLessThanMaxDeletes(): Unit = {
+ val runner = new FakeRunner(Seq(success(Row(550L)), success(Row(550L)), success(Row(20L))))
+ val result =
+ new PaimonExpireSnapshotsAction(runner)
+ .execute(context(), table, defaultConfig)
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals(3, runner.sqls.size)
+ assertEquals(Seq("success", "success", "success"), runner.outcomes)
+ assertEquals(Seq("round-1", "round-2", "round-3"), result.tasks.map(_.name))
+ assertEquals(
+ Seq(ActionStatus.Success, ActionStatus.Success, ActionStatus.Success),
+ result.tasks.map(_.status))
+ assertEquals("3", result.metrics("rounds"))
+ assertEquals("1120", result.metrics("totalDeletedCount"))
+ assertEquals("1120", result.metrics("deletedSnapshotsTotal"))
+ assertEquals("550", result.metrics("maxDeletes"))
+ }
+
+ @Test
+ def executeRetriesOnlyTheCurrentRoundWhenSqlFails(): Unit = {
+ val runner =
+ new FakeRunner(
+ Seq(
+ success(Row(550L)),
+ failure(new RuntimeException("round 2 failed")),
+ success(Row(20L))))
+ val result =
+ new PaimonExpireSnapshotsAction(runner)
+ .execute(context(retryTimes = 2), table, defaultConfig)
+
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals(3, runner.sqls.size)
+ assertEquals(Seq("success", "failure", "success"), runner.outcomes)
+ assertEquals(Seq("round-1", "round-2"), result.tasks.map(_.name))
+ assertEquals("1", result.tasks.head.metrics("attempts"))
+ assertEquals("2", result.tasks(1).metrics("attempts"))
+ }
+
+ @Test
+ def executeThrowsWhenCurrentRoundRetryIsExhausted(): Unit = {
+ val runner =
+ new FakeRunner(
+ Seq(
+ success(Row(550L)),
+ failure(new RuntimeException("round 2 failed 1")),
+ failure(new RuntimeException("round 2 failed 2"))))
+
+ assertThrows(
+ classOf[RuntimeException],
+ () =>
+ new PaimonExpireSnapshotsAction(runner)
+ .execute(context(retryTimes = 2), table, defaultConfig))
+ assertEquals(3, runner.sqls.size)
+ assertEquals(Seq("success", "failure", "failure"), runner.outcomes)
+ }
+
+ @Test
+ def executeParsesDeletedCountFromKnownFieldsAndFirstColumnFallback(): Unit = {
+ assertEquals(
+ "1",
+ executeOnce(
+ new GenericRowWithSchema(
+ Array[Any](1L),
+ StructType(Seq(StructField("deleted_count", LongType))))).metrics("totalDeletedCount"))
+ assertEquals(
+ "1",
+ executeOnce(
+ new GenericRowWithSchema(
+ Array[Any](1L),
+ StructType(Seq(StructField("deleted_count", LongType))))).metrics(
+ "deletedSnapshotsTotal"))
+ assertEquals(
+ "2",
+ executeOnce(
+ new GenericRowWithSchema(
+ Array[Any](2L),
+ StructType(Seq(StructField("deleted_snapshots", LongType))))).metrics(
+ "totalDeletedCount"))
+ assertEquals(
+ "3",
+ executeOnce(
+ new GenericRowWithSchema(
+ Array[Any](3L),
+ StructType(Seq(StructField("result", LongType))))).metrics("totalDeletedCount"))
+ assertEquals(
+ "4",
+ executeOnce(
+ new GenericRowWithSchema(
+ Array[Any](4L),
+ StructType(Seq(StructField("deleted_snapshots_count", LongType))))).metrics(
+ "totalDeletedCount"))
+ assertEquals("5", executeOnce(Row(5L)).metrics("totalDeletedCount"))
+ }
+
+ private def executeOnce(row: Row): org.apache.amoro.producer.core.ActionResult = {
+ new PaimonExpireSnapshotsAction(new FakeRunner(Seq(success(row))))
+ .execute(context(), table, defaultConfig)
+ }
+
+ private def parse(
+ action: PaimonExpireSnapshotsAction,
+ args: Array[String] = Array.empty): PaimonExpireSnapshotsConfig = {
+ val commandLine = ProducerCliParser.parseMerged(action.options, args)
+ action.parse(commandLine).asInstanceOf[PaimonExpireSnapshotsConfig]
+ }
+
+ private def context(retryTimes: Int = 3): ProducerContext = {
+ ProducerContext(null.asInstanceOf[SparkSession], commonConfig(retryTimes), DummyAdapter)
+ }
+
+ private def commonConfig(retryTimes: Int = 3): CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "expire-snapshots",
+ catalogName = "paimon",
+ databaseName = Some("db"),
+ tableName = Some("t1"),
+ tableNameRegex = None,
+ retryTimes = retryTimes,
+ continueOnTableFailure = true)
+ }
+
+ private val table = TargetTable("db.t1", "db", "t1")
+
+ private object WrongConfig extends ActionConfig
+
+ private val defaultConfig =
+ PaimonExpireSnapshotsConfig(
+ retainMax = 20,
+ retainMin = 10,
+ maxDeletes = 550L,
+ procedureOptions = "file-operation.thread-num=32,snapshot.expire.execution-mode=sync")
+
+ private object DummyAdapter extends LakeFormatAdapter {
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = Seq.empty
+ }
+
+ private def success(row: Row): Either[RuntimeException, Row] = Right(row)
+
+ private def failure(error: RuntimeException): Either[RuntimeException, Row] = Left(error)
+
+ private class FakeRunner(results: Seq[Either[RuntimeException, Row]]) extends PaimonSqlRunner {
+
+ var sqls: Seq[String] = Seq.empty
+ var outcomes: Seq[String] = Seq.empty
+ private var remainingResults = results
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ sqls :+= sqlText
+ if (remainingResults.isEmpty) {
+ throw new AssertionError("FakeRunner result queue exhausted")
+ }
+ val nextResult = remainingResults.head
+ remainingResults = remainingResults.tail
+ nextResult match {
+ case Left(error) =>
+ outcomes :+= "failure"
+ throw error
+ case Right(row) =>
+ outcomes :+= "success"
+ Seq(row)
+ }
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapterTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapterTest.scala
new file mode 100644
index 000000000..82c9f45ba
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonFormatAdapterTest.scala
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{CommonProducerConfig, ProducerApp, ProducerCliParser, ProducerContext, TargetTable}
+
+class PaimonFormatAdapterTest {
+
+ @Test
+ def adapterExposesFourActions(): Unit = {
+ assertEquals("paimon", PaimonFormatAdapter.format)
+ assertEquals(
+ Seq("compact", "compact-manifest", "expire-snapshots", "remove-orphan-files"),
+ PaimonFormatAdapter.actions.map(_.name))
+ }
+
+ @Test
+ def adapterActionsUseInjectedRunner(): Unit = {
+ val runner = new FakeRunner(Seq(Row("1")))
+ val adapter = new PaimonFormatAdapter(runner)
+ val context = ProducerContext(null, commonConfig.copy(retryTimes = 1), adapter)
+ val table = TargetTable("db.orders", "db", "orders")
+
+ adapter.actions.foreach { action =>
+ val commandLine = ProducerCliParser.parseMerged(action.options, Array.empty[String])
+ action.execute(context, table, action.parse(commandLine))
+ }
+
+ assertTrue(runner.sqls.exists(_.contains("sys.compact(")))
+ assertTrue(runner.sqls.exists(_.contains("sys.compact_manifest(")))
+ assertTrue(runner.sqls.exists(_.contains("sys.expire_snapshots(")))
+ assertTrue(runner.sqls.exists(_.contains("sys.remove_orphan_files(")))
+ }
+
+ @Test
+ def producerAppResolvesRealPaimonAdapterAndAction(): Unit = {
+ val adapter = ProducerApp.resolveAdapter("paimon", Seq(PaimonFormatAdapter))
+ val action = ProducerApp.resolveAction(PaimonFormatAdapter, "compact")
+
+ assertEquals("paimon", adapter.format)
+ assertEquals("compact", action.name)
+ }
+
+ @Test
+ def producerAppRejectsMissingRealPaimonActionWithAvailableActions(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => ProducerApp.resolveAction(PaimonFormatAdapter, "missing"))
+
+ assertTrue(exception.getMessage.contains("compact"))
+ assertTrue(exception.getMessage.contains("compact-manifest"))
+ assertTrue(exception.getMessage.contains("expire-snapshots"))
+ assertTrue(exception.getMessage.contains("remove-orphan-files"))
+ }
+
+ @Test
+ def producerAppWithRealPaimonAdapterRejectsMissingFormatOrActionBeforeSpark(): Unit = {
+ var sparkCreated = false
+
+ val exitCode =
+ ProducerApp.runInternal(
+ Array("--databaseName", "db"),
+ Seq(PaimonFormatAdapter),
+ () => {
+ sparkCreated = true
+ null
+ })
+
+ assertEquals(2, exitCode)
+ assertTrue(!sparkCreated)
+ }
+
+ @Test
+ def producerAppWithRealPaimonAdapterRejectsUnsupportedFormatOrActionBeforeSpark(): Unit = {
+ var sparkCreated = false
+ val builder = () => {
+ sparkCreated = true
+ null
+ }
+
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "iceberg", "--action", "compact", "--databaseName", "db"),
+ Seq(PaimonFormatAdapter),
+ builder))
+ assertEquals(
+ 2,
+ ProducerApp.runInternal(
+ Array("--format", "paimon", "--action", "missing", "--databaseName", "db"),
+ Seq(PaimonFormatAdapter),
+ builder))
+ assertTrue(!sparkCreated)
+ }
+
+ @Test
+ def listTablesFiltersTemporaryTables(): Unit = {
+ val runner =
+ new FakeRunner(
+ Seq(
+ Row("db", "orders", false),
+ Row("db", "tmp_orders", true),
+ Row("db", "customers", "false"),
+ Row("db", "tmp_customers", "1")))
+ val adapter = new PaimonFormatAdapter(runner)
+
+ val targets = adapter.listTables(null, commonConfig, "db")
+
+ assertEquals(
+ Seq(TargetTable("db.orders", "db", "orders"), TargetTable("db.customers", "db", "customers")),
+ targets)
+ assertEquals(Seq("SHOW TABLES IN `paimon`.`db`"), runner.sqls)
+ }
+
+ @Test
+ def listTablesAcceptsTableColumnFallback(): Unit = {
+ val runner = new FakeRunner(Seq(Row("database_ignored", "events", false)))
+ val adapter = new PaimonFormatAdapter(runner)
+
+ val targets = adapter.listTables(null, commonConfig, "db")
+
+ assertEquals(Seq(TargetTable("db.events", "db", "events")), targets)
+ }
+
+ @Test
+ def listTablesIgnoresEmptyTableNames(): Unit = {
+ val runner = new FakeRunner(Seq(Row("db", " ", false), Row("db", null, false)))
+ val adapter = new PaimonFormatAdapter(runner)
+
+ assertTrue(adapter.listTables(null, commonConfig, "db").isEmpty)
+ }
+
+ private def commonConfig: CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "compact",
+ catalogName = "paimon",
+ databaseName = Some("db"),
+ tableName = None,
+ tableNameRegex = None,
+ retryTimes = 3,
+ continueOnTableFailure = true)
+ }
+
+ final private class FakeRunner(rows: Seq[Row]) extends PaimonSqlRunner {
+ var sqls: Seq[String] = Seq.empty
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ sqls = sqls :+ sqlText
+ rows
+ }
+ }
+}
diff --git a/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesActionTest.scala b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesActionTest.scala
new file mode 100644
index 000000000..ddf40aaed
--- /dev/null
+++ b/amoro-producer/src/test/scala/org/apache/amoro/producer/paimon/PaimonRemoveOrphanFilesActionTest.scala
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.producer.paimon
+
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema
+import org.apache.spark.sql.types.{LongType, StructField, StructType}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertThrows, assertTrue}
+import org.junit.jupiter.api.Test
+
+import org.apache.amoro.producer.core.{ActionStatus, CommonProducerConfig, ProducerCliParser, ProducerContext, RetryMode, TargetTable}
+import org.apache.amoro.producer.spi.{ActionConfig, LakeFormatAdapter, ProducerAction}
+
+class PaimonRemoveOrphanFilesActionTest {
+
+ @Test
+ def parseUsesDefaultsAndTrimsMode(): Unit = {
+ val action = new PaimonRemoveOrphanFilesAction()
+
+ assertEquals("remove-orphan-files", action.name)
+ assertEquals(RetryMode.FrameworkTableRetry, action.retryMode)
+ assertEquals(PaimonRemoveOrphanFilesConfig(10, "distributed"), parse(action))
+ assertEquals(
+ PaimonRemoveOrphanFilesConfig(2, "local"),
+ parse(action, Array("--parallelism", " 2 ", "--mode", " local ")))
+ }
+
+ @Test
+ def validateRejectsInvalidArguments(): Unit = {
+ val action = new PaimonRemoveOrphanFilesAction()
+ val common = commonConfig()
+
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonRemoveOrphanFilesConfig(0, "distributed")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonRemoveOrphanFilesConfig(-1, "distributed")))
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => action.validate(common, PaimonRemoveOrphanFilesConfig(10, " ")))
+ }
+
+ @Test
+ def validateRejectsWrongActionConfigType(): Unit = {
+ val exception =
+ assertThrows(
+ classOf[IllegalArgumentException],
+ () => new PaimonRemoveOrphanFilesAction().validate(commonConfig(), WrongConfig))
+
+ assertTrue(exception.getMessage.contains("remove-orphan-files"))
+ assertTrue(exception.getMessage.contains("PaimonRemoveOrphanFilesConfig"))
+ }
+
+ @Test
+ def buildSqlIncludesCatalogTableParallelismAndEscapedMode(): Unit = {
+ val sql =
+ PaimonRemoveOrphanFilesAction.buildRemoveOrphanFilesSql(
+ catalogName = "pa`imon",
+ config = PaimonRemoveOrphanFilesConfig(10, "dis'tributed"),
+ table = table)
+
+ assertTrue(sql.contains("CALL `pa``imon`.sys.remove_orphan_files("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("parallelism => 10"))
+ assertTrue(sql.contains("mode => 'dis''tributed'"))
+ }
+
+ @Test
+ def buildSqlUsesDefaultParallelismAndMode(): Unit = {
+ val sql =
+ PaimonRemoveOrphanFilesAction.buildRemoveOrphanFilesSql(
+ catalogName = "paimon",
+ config = PaimonRemoveOrphanFilesConfig(10, "distributed"),
+ table = table)
+
+ assertTrue(sql.contains("CALL `paimon`.sys.remove_orphan_files("))
+ assertTrue(sql.contains("table => 'db.t1'"))
+ assertTrue(sql.contains("parallelism => 10"))
+ assertTrue(sql.contains("mode => 'distributed'"))
+ }
+
+ @Test
+ def executeAggregatesMetricsFromAllReturnedRows(): Unit = {
+ val config = PaimonRemoveOrphanFilesConfig(5, "local")
+ val runner =
+ new FakeRunner(
+ Seq(
+ new GenericRowWithSchema(
+ Array[Any](1L, 10L),
+ StructType(
+ Seq(
+ StructField("deleted_file_count", LongType),
+ StructField("deleted_file_total_len_in_bytes", LongType)))),
+ Row("2", "20")))
+ val result =
+ new PaimonRemoveOrphanFilesAction(runner)
+ .execute(context(), table, config)
+
+ assertEquals(1, runner.sqls.size)
+ assertTrue(runner.sqls.head.contains("CALL `paimon`.sys.remove_orphan_files("))
+ assertTrue(runner.sqls.head.contains("table => 'db.t1'"))
+ assertTrue(runner.sqls.head.contains("parallelism => 5"))
+ assertTrue(runner.sqls.head.contains("mode => 'local'"))
+ assertEquals(ActionStatus.Success, result.status)
+ assertEquals("3", result.metrics("deletedFileCount"))
+ assertEquals("30", result.metrics("deletedFileTotalLenInBytes"))
+ assertEquals(Seq(ActionStatus.Success), result.tasks.map(_.status))
+ assertEquals(Seq("remove_orphan_files"), result.tasks.map(_.name))
+ assertTrue(result.costMs >= 0.0)
+ }
+
+ private def parse(
+ action: PaimonRemoveOrphanFilesAction,
+ args: Array[String] = Array.empty): PaimonRemoveOrphanFilesConfig = {
+ val commandLine = ProducerCliParser.parseMerged(action.options, args)
+ action.parse(commandLine).asInstanceOf[PaimonRemoveOrphanFilesConfig]
+ }
+
+ private def context(): ProducerContext = {
+ ProducerContext(null.asInstanceOf[SparkSession], commonConfig(), DummyAdapter)
+ }
+
+ private def commonConfig(): CommonProducerConfig = {
+ CommonProducerConfig(
+ format = "paimon",
+ action = "remove-orphan-files",
+ catalogName = "paimon",
+ databaseName = Some("db"),
+ tableName = Some("t1"),
+ tableNameRegex = None,
+ retryTimes = 3,
+ continueOnTableFailure = true)
+ }
+
+ private val table = TargetTable("db.t1", "db", "t1")
+
+ private object WrongConfig extends ActionConfig
+
+ private object DummyAdapter extends LakeFormatAdapter {
+ override def format: String = "paimon"
+
+ override def actions: Seq[ProducerAction] = Seq.empty
+
+ override def listTables(
+ spark: SparkSession,
+ config: CommonProducerConfig,
+ database: String): Seq[TargetTable] = Seq.empty
+ }
+
+ private class FakeRunner(rows: Seq[Row]) extends PaimonSqlRunner {
+ var sqls: Seq[String] = Seq.empty
+
+ override def run(spark: SparkSession, sqlText: String): Seq[Row] = {
+ sqls :+= sqlText
+ rows
+ }
+ }
+}