You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "zhangwl9 (via GitHub)" <gi...@apache.org> on 2026/05/07 09:24:45 UTC

[PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

zhangwl9 opened a new pull request, #4209:
URL: https://github.com/apache/amoro/pull/4209

   # Conflicts:
   #	amoro-common/src/test/java/org/apache/amoro/process/TestLocalExecutionEngine.java
   
   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
     2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
   -->
   
   ## Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about its use case.
     2. If you fix a bug, you can clarify why it is a bug.
     3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
   -->
   
   Close #4208.
   
   ## Brief change log
   <!--
   Clearly describe the changes made in modules, classes, methods, etc.
   -->
   
   -
   
   ## How was this patch tested?
   
   - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [x] Run test locally before making a pull request
   
   ## Documentation
   
   - Does this pull request introduce a new feature? (yes / no)
   - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhangwl9 (via GitHub)" <gi...@apache.org>.
zhangwl9 commented on PR #4209:
URL: https://github.com/apache/amoro/pull/4209#issuecomment-4402524830

   @baiyangtx  @xxubai  Could you help me review it when you are free? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #4209:
URL: https://github.com/apache/amoro/pull/4209#discussion_r3233133950


##########
amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/OrphanFilesCleaningProcess.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.process.iceberg;
+
+import org.apache.amoro.Action;
+import org.apache.amoro.AmoroTable;
+import org.apache.amoro.IcebergActions;
+import org.apache.amoro.TableRuntime;
+import org.apache.amoro.maintainer.TableMaintainer;
+import org.apache.amoro.process.ExecuteEngine;
+import org.apache.amoro.process.LocalProcess;
+import org.apache.amoro.process.TableProcess;
+import org.apache.amoro.server.optimizing.maintainer.TableMaintainerFactory;
+import org.apache.amoro.server.table.DefaultTableRuntime;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+/** Local table process for cleaning Iceberg orphan files. */
+public class OrphanFilesCleaningProcess extends TableProcess implements LocalProcess {
+
+  private static final Logger LOG = LoggerFactory.getLogger(OrphanFilesCleaningProcess.class);
+
+  public OrphanFilesCleaningProcess(TableRuntime tableRuntime, ExecuteEngine engine) {
+    super(tableRuntime, engine);
+  }
+
+  @Override
+  public String tag() {
+    return getAction().getName().toLowerCase();
+  }
+
+  @Override
+  public void run() {
+    try {
+      AmoroTable<?> amoroTable = tableRuntime.loadTable();
+      TableMaintainer tableMaintainer = TableMaintainerFactory.create(amoroTable, tableRuntime);
+      tableMaintainer.cleanOrphanFiles();
+      tableRuntime.updateState(

Review Comment:
   The old `OrphanFilesCleaningExecutor` introduced a random startup jitter via `getExecutorDelay()`:
   
   ```java
   // OrphanFilesCleaningExecutor.java
   @Override
   protected long getExecutorDelay() {
       return ThreadLocalRandom.current().nextLong(interval.toMillis());
   }
   ```
   
   This was intentional — with many tables registered, a fixed `executorDelay` causes all tables to be evaluated (and potentially trigger cleanup) at nearly the same time, creating a thundering-herd on the underlying storage.
   
   In the new framework, `DefaultActionCoordinator.getExecutorDelay()` always returns the full interval:
   
   ```java
   @Override
   public long getExecutorDelay() {
       return strategy.getTriggerInterval().toMillis();
   }
   ```
   
   This is a framework-level behavior (shared with `SnapshotsExpiringProcess`), so fixing it here may be out of scope. But it is worth tracking — either by adding jitter support to `ProcessTriggerStrategy`, or by overriding `getExecutorDelay()` in `DefaultActionCoordinator` with a random offset bounded by the interval.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #4209:
URL: https://github.com/apache/amoro/pull/4209#discussion_r3233140142


##########
amoro-ams/src/main/java/org/apache/amoro/server/scheduler/inline/InlineTableExecutors.java:
##########
@@ -42,13 +41,6 @@ public static InlineTableExecutors getInstance() {
   }

Review Comment:
   The old global configs in `AmoroManagementConf` are still present after this refactor:
   
   ```java
   // AmoroManagementConf.java
   public static final ConfigOption<Boolean> CLEAN_ORPHAN_FILES_ENABLED = ...
   public static final ConfigOption<Integer> CLEAN_ORPHAN_FILES_THREAD_COUNT = ...
   public static final ConfigOption<Duration> CLEAN_ORPHAN_FILES_INTERVAL = ...
   ```
   
   And `AmoroManagementConfValidator` still validates them. Since the configuration has moved to `process-factories.yaml`, these entries are now dead code. They should either be removed or marked `@Deprecated` with a note pointing to the new config location, to avoid confusing users who upgrade and wonder why their old `ams.yaml` settings are silently ignored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhangwl9 (via GitHub)" <gi...@apache.org>.
zhangwl9 commented on code in PR #4209:
URL: https://github.com/apache/amoro/pull/4209#discussion_r3239337995


##########
amoro-ams/src/main/java/org/apache/amoro/server/scheduler/inline/InlineTableExecutors.java:
##########
@@ -42,13 +41,6 @@ public static InlineTableExecutors getInstance() {
   }

Review Comment:
   Thanks for the review! The old CLEAN_ORPHAN_FILES_* configs in AmoroManagementConf and their validation in AmoroManagementConfValidator have already been removed in the follow-up commit . The configuration has been fully migrated to IcebergProcessFactory (process-factories.yaml) and documented in the deployment guide.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #4209:
URL: https://github.com/apache/amoro/pull/4209


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-4208] Refactor orphan-files-cleaning via ProcessFactory plugin [amoro]

Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #4209:
URL: https://github.com/apache/amoro/pull/4209#discussion_r3233133950


##########
amoro-ams/src/main/java/org/apache/amoro/server/process/iceberg/OrphanFilesCleaningProcess.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.process.iceberg;
+
+import org.apache.amoro.Action;
+import org.apache.amoro.AmoroTable;
+import org.apache.amoro.IcebergActions;
+import org.apache.amoro.TableRuntime;
+import org.apache.amoro.maintainer.TableMaintainer;
+import org.apache.amoro.process.ExecuteEngine;
+import org.apache.amoro.process.LocalProcess;
+import org.apache.amoro.process.TableProcess;
+import org.apache.amoro.server.optimizing.maintainer.TableMaintainerFactory;
+import org.apache.amoro.server.table.DefaultTableRuntime;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+/** Local table process for cleaning Iceberg orphan files. */
+public class OrphanFilesCleaningProcess extends TableProcess implements LocalProcess {
+
+  private static final Logger LOG = LoggerFactory.getLogger(OrphanFilesCleaningProcess.class);
+
+  public OrphanFilesCleaningProcess(TableRuntime tableRuntime, ExecuteEngine engine) {
+    super(tableRuntime, engine);
+  }
+
+  @Override
+  public String tag() {
+    return getAction().getName().toLowerCase();
+  }
+
+  @Override
+  public void run() {
+    try {
+      AmoroTable<?> amoroTable = tableRuntime.loadTable();
+      TableMaintainer tableMaintainer = TableMaintainerFactory.create(amoroTable, tableRuntime);
+      tableMaintainer.cleanOrphanFiles();
+      tableRuntime.updateState(

Review Comment:
   The old `OrphanFilesCleaningExecutor` introduced a random startup jitter via `getExecutorDelay()`:
   
   ```java
   // OrphanFilesCleaningExecutor.java
   @Override
   protected long getExecutorDelay() {
       return ThreadLocalRandom.current().nextLong(interval.toMillis());
   }
   ```
   
   This was intentional — with many tables registered, a fixed `executorDelay` causes all tables to be evaluated (and potentially trigger cleanup) at nearly the same time, creating a thundering-herd on the underlying storage.
   
   In the new framework, `DefaultActionCoordinator.getExecutorDelay()` always returns the full interval:
   
   ```java
   @Override
   public long getExecutorDelay() {
       return strategy.getTriggerInterval().toMillis();
   }
   ```
   
   This is a framework-level behavior (shared with `SnapshotsExpiringProcess`), so fixing it here may be out of scope. But it is worth tracking — either by adding jitter support to `ProcessTriggerStrategy`, or by overriding `getExecutorDelay()` in `DefaultActionCoordinator` with a random offset bounded by the interval.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org