You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by ji...@apache.org on 2025/08/25 06:12:24 UTC
(amoro) branch 0.8.x updated (7d2b88c99 -> 56b686abb)
This is an automated email from the ASF dual-hosted git repository.
jinsongzhou pushed a change to branch 0.8.x
in repository https://gitbox.apache.org/repos/asf/amoro.git
from 7d2b88c99 Merge branch 'master' into 0.8.x
new 5a6d1c207 [Bug]: The flink optimizer often triggers OOM exceptions. In fact, the files processed by each task are not large. (#3521)
new 9d67b3009 [AMORO-3533] Avoid bulk table removal when synchronizing external catalogs and fetching database table listings fails (#3534)
new 2d2664179 (hotfix) enhance log for iceberg table maintainer (#3543)
new ccbaf466e fix some doc error (#3551)
new 96b14bb71 [AMORO-3538] Resolve Hive 4 Metastore listing failure on Hadoop2 (#3539)
new 33aa81566 [AMORO-3559] Extract abstract class for table metrics (#3560)
new b377f3842 [hotfix]update upgrade.sql (#3563)
new 2e179bf0b [Hotfix] Fix HudiTableDesctiptor error (#3572)
new 87908da19 [AMORO-3549] Fix AMS helm chart cannot set log4j2 because of wrong indentation (#3550)
new 1ad20d99f [Hotfix] update upgrade.sql for postgres after 0.8.0 release. (#3574)
new d655d2e45 [AMORO-3540] Increase the precision of optimizing_status_start_time from s to ms. (#3542)
new 183635b59 Optimize the performance of optimizer pollTask (#3583)
new ca00dcdfb [AMORO-3578] Improving Error Prompts for Login Failures (#3579)
new 8790b741c [Hotfix] Bump parquet-avro to 1.15.2 to fix CVE (#3580)
new 3b740c4fc [AMORO-3545] Prevent empty tasks list during planning (#3546)
new 8b53d39e0 Optimize log output for SchedulingPolicy (#3592)
new dae0e589f [AMORO-3594] Improve table identifiers display (#3595)
new f660d3c99 [AMORO-3587] Modify the default value of hive catalog clients to 20 (#3590)
new 8fce30182 [AMORO-3599] remove com.esotericsoftware.kryo dependencies in flink optimizer package (#3600)
new 8c75c62d2 [AMORO-3299] Fix the problem that the commit time of the file is displayed incorrectly (#3604)
new 3ac36310a [AMORO-3570] Add Rest Catalog configuration documentation (#3597)
new 61f952703 [hotfix] Throw the exception when the tableRuntime disposal fails. (#3557)
new 5c504d1a6 [AMORO-3616][Improvement] Correct the spelling errors in doc of Managing Optimizers page (#3618)
new 85c5eaa6e [AMORO-3612][Improvement]: Add the log output function of the StandaloneOptimizer module (#3613)
new 5e777dc89 [AMORO-3614][Improvement]: Add a document about deploying Amoro in IDE (#3615)
new c814c06dc [AMORO-3620][Improvement]: Fix the error in the calculation of the health score of table (#3621)
new a736fa797 [AMORO-3622] Fix expiring data when file without upperbounds (#3623)
new 76cbaa7fd [AMORO-3633][improvement]: Fix the sorting rules of topTables order by healthScore (#3634)
new 9d504d01d [Hotfix] Handle null partition values in IcebergTableMaintainer to prevent expiration errors (#3635)
new 347b399fe [AMORO-3631][Improvement]: Update the doc for optimizer management (#3641)
new fdff0b398 [AMORO-3645][Improvement]: Optimize target file size after self-optimizing (#3646)
new 2a229b459 [AMORO-3664] Trim the leading and trailing whitespace in ResourceGroup properties (#3664)
new 01cc1de8d Fixed the memory display error in Spark optimizer resource allocation.
new 49a959280 [AMORO-3637] Harden the task completion logic to avoid the task state in ACKED forever
new cdd250b91 [Hotfix] Replace Jersey Guava imports with Amoro shaded Guava (#3674)
new f1a36194d Fix continually creating unused keep_file when committing for an unpartitioned mixed_hive table optimization (#3690)
new 717d09a09 [AMORO-2519] Improve the display of Duration in the page of Optimizing>Tables to make it more user-friendly (#3694)
new af986791d [AMORO-3638] Add support for task retries when task execution times out
new e451f7389 [Improvement] Use fileIndex="nomax" to avoid log loss. (#3701)
new 36ee3f4c9 [AMORO-3245] Display comment on Table page. (#3672)
new dcfb3cb71 [AMORO-3703] Use dataFileFormat for deleteFile if not set (#3704)
new fcd539654 [AMORO-3682]Fix table creation failure when comment contains `PRIMARY KEY` text (#3683)
new fb2752059 [AMORO-3695][Improvement]: Reduce unnecessary details in the log output (#3696)
new af2ec1263 [AMORO-3677] Fill catalogProperties in creating catalog rather than building (#3678)
new 90929ca78 [AMORO-3685] Optimize the performance of fillSkipSet for planning (#3688)
new 2a7c690b5 [AMORO-3718]Inconsistency in capitalization between iceberg and hive … (#3719)
new 04bd1a96c [AMORO-3716]Enhance FlinkUnifiedCatalog getTable efficiency (#3717)
new 078d7ec2d [AMORO-3720] fix mixed_format does not support explain for merge into SQL (#3721)
new a7b3f15fc [AMORO-3713] fix bug of merging into when selecting less columns from souce table than target mixed table (#3722)
new beddc84b4 [AMORO-3732] Upgrade commons-beanutils version to 1.11.0 (#3733)
new de4fd4c96 [AMORO-3739][Bug]: Add iceberg-arrow to ams package (#3740)
new 56b686abb [hotfix] Fix doc error in displaying "Using Customized Encryption Method for Configurations" on admin guidance page (#3741)
The 52 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CONTRIBUTING.md | 86 ++++++
amoro-ams/pom.xml | 9 +
.../apache/amoro/server/AmoroManagementConf.java | 6 +
.../amoro/server/DefaultOptimizingService.java | 31 +-
.../amoro/server/catalog/CatalogBuilder.java | 11 -
.../server/catalog/DefaultCatalogManager.java | 27 ++
.../dashboard/MixedAndIcebergTableDescriptor.java | 12 +-
.../dashboard/controller/LoginController.java | 2 +-
.../controller/OptimizerGroupController.java | 5 +-
.../dashboard/controller/OverviewController.java | 2 +-
.../dashboard/controller/SettingController.java | 2 +-
.../server/dashboard/utils/PropertiesUtil.java | 32 +++
.../amoro/server/optimizing/KeyedTableCommit.java | 18 +-
.../amoro/server/optimizing/OptimizingQueue.java | 54 +++-
.../amoro/server/optimizing/SchedulingPolicy.java | 4 +-
.../server/optimizing/UnKeyedTableCommit.java | 11 +-
.../maintainer/IcebergTableMaintainer.java | 57 ++--
.../persistence/mapper/TableBlockerMapper.java | 23 ++
.../amoro/server/table/AbstractTableMetrics.java | 71 +++++
.../amoro/server/table/DefaultTableService.java | 34 ++-
.../amoro/server/table/TableOptimizingMetrics.java | 40 +--
.../table/TableOrphanFilesCleaningMetrics.java | 42 +--
.../apache/amoro/server/table/TableRuntime.java | 4 +-
.../amoro/server/table/TableSummaryMetrics.java | 38 +--
.../amoro/server/table/blocker/TableBlocker.java | 7 +-
.../src/main/resources/mysql/ams-mysql-init.sql | 2 +-
.../{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} | 3 -
amoro-ams/src/main/resources/mysql/upgrade.sql | 61 +---
.../main/resources/postgres/ams-postgres-init.sql | 2 +-
.../{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} | 0
amoro-ams/src/main/resources/postgres/upgrade.sql | 108 +------
.../apache/amoro/server/AMSServiceTestBase.java | 2 +
.../amoro/server/TestDefaultOptimizingService.java | 30 ++
.../server/dashboard/utils/TestPropertiesUtil.java | 42 +++
.../amoro/server/table/TestCatalogService.java | 12 +
.../table/TestSyncTableOfExternalCatalog.java | 320 +++++++++++++++++++++
amoro-common/pom.xml | 5 +
.../amoro/properties/CatalogMetaProperties.java | 2 +-
.../amoro/table/descriptor/ServerTableMeta.java | 16 +-
.../amoro/formats/hudi/HudiTableDescriptor.java | 10 +-
...he.amoro.table.descriptor.FormatTableDescriptor | 2 +-
amoro-format-iceberg/pom.xml | 5 -
.../optimizing/AbstractRewriteFilesExecutor.java | 4 +-
.../amoro/optimizing/IcebergRewriteExecutor.java | 13 +
.../plan/AbstractOptimizingEvaluator.java | 2 +-
.../optimizing/plan/AbstractOptimizingPlanner.java | 36 ++-
.../optimizing/plan/CommonPartitionEvaluator.java | 4 +-
.../org/apache/amoro/table/TableProperties.java | 3 +
.../amoro/flink/catalog/FlinkUnifiedCatalog.java | 57 ++--
.../apache/amoro/flink/catalog/MixedCatalog.java | 31 +-
.../org/apache/amoro/hive/op/UpdateHiveFiles.java | 20 +-
.../amoro/hive/utils/HiveMetaSynchronizer.java | 36 ++-
.../v3.2/amoro-mixed-spark-3.2/pom.xml | 4 +
.../v3.3/amoro-mixed-spark-3.3/pom.xml | 4 +
.../RewriteMixedFormatMergeIntoTable.scala | 13 +-
.../parser/MixedFormatSqlExtensionsParser.scala | 3 +
.../parser/MixedFormatSqlExtendAstBuilder.scala | 6 +-
.../spark/test/suites/sql/TestCreateTableSQL.java | 97 +++++++
.../v3.5/amoro-mixed-spark-3.5/pom.xml | 4 +
.../parser/MixedFormatSqlExtendAstBuilder.scala | 6 +-
.../spark/test/suites/sql/TestCreateTableSQL.java | 97 +++++++
.../apache/amoro/trino/mixed/TestKeyedTable.java | 2 +-
.../formats/paimon/PaimonTableDescriptor.java | 3 +
.../amoro/optimizer/common/OptimizerExecutor.java | 33 ++-
amoro-optimizer/amoro-optimizer-flink/pom.xml | 2 +-
.../amoro/optimizer/flink/FlinkExecutor.java | 18 +-
.../amoro/optimizer/spark/SparkOptimizer.java | 5 +-
amoro-optimizer/amoro-optimizer-standalone/pom.xml | 33 +++
.../optimizer/standalone/StandaloneOptimizer.java | 1 +
.../src/main/resources/log4j2.xml | 13 -
amoro-web/src/types/common.type.ts | 1 +
amoro-web/src/views/optimize/components/List.vue | 6 +-
amoro-web/src/views/tables/components/Details.vue | 4 +-
amoro-web/src/views/tables/components/Files.vue | 2 +-
amoro-web/src/views/tables/index.vue | 6 +-
charts/amoro/templates/_pod.tpl | 2 +-
charts/amoro/tests/amoro-deployment_test.yaml | 10 +-
dist/src/main/amoro-bin/conf/config.yaml | 1 +
dist/src/main/amoro-bin/conf/log4j2.xml | 13 +-
dist/src/main/amoro-bin/conf/optimize/log4j2.xml | 7 +-
docs/_index.md | 2 +-
docs/admin-guides/deployment-on-kubernetes.md | 15 +-
docs/admin-guides/deployment.md | 3 +-
docs/admin-guides/managing-catalogs.md | 11 +
docs/admin-guides/managing-optimizers.md | 20 +-
...tomized-encryption-method-for-configurations.md | 8 +-
docs/engines/flink/flink-cdc-ingestion.md | 4 +-
docs/engines/flink/flink-dml.md | 14 +-
docs/engines/flink/flink-get-started.md | 2 +-
docs/images/admin/create-rest_catalog.png | Bin 0 -> 49616 bytes
docs/images/admin/optimizer_create.png | Bin 0 -> 54835 bytes
docs/images/admin/optimizer_scale.png | Bin 484488 -> 0 bytes
docs/user-guides/cdc-ingestion.md | 4 +-
pom.xml | 26 +-
tools/maven/checkstyle.xml | 5 +
95 files changed, 1439 insertions(+), 527 deletions(-)
create mode 100644 amoro-ams/src/main/java/org/apache/amoro/server/table/AbstractTableMetrics.java
copy amoro-ams/src/main/resources/mysql/{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} (96%)
copy amoro-ams/src/main/resources/postgres/{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} (100%)
copy {amoro-ams => amoro-optimizer/amoro-optimizer-standalone}/src/main/resources/log4j2.xml (71%)
create mode 100644 docs/images/admin/create-rest_catalog.png
create mode 100644 docs/images/admin/optimizer_create.png
delete mode 100644 docs/images/admin/optimizer_scale.png