[GitHub] [incubator-doris] pengxiangyu commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly

[GitBox <gi...@apache.org>]

pengxiangyu commented on a change in pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r745579688



##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java
##########
@@ -73,5 +73,7 @@ private void process() throws UserException {
                 job.cancel(msgType, exception.getMessage());
             }
         }
+
+        this.syncJobManager.cleanOldSyncJobs();

Review comment:
       cleanOldSyncJobs() 放在runAfterCatalogReady()里更好些，新建与删除最好分开在两个函数。

##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java
##########
@@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException {
         }
     }
 
+    // Remove old sync jobs. Called periodically.
+    // Stopped jobs will be removed after Config.label_keep_max_second.
+    public void cleanOldSyncJobs() {
+        LOG.debug("begin to clean old sync jobs ");
+        long currentTimeMs = System.currentTimeMillis();
+        writeLock();
+        try {
+            Iterator<Map.Entry<Long, SyncJob>> iterator = idToSyncJob.entrySet().iterator();
+            while (iterator.hasNext()) {
+                SyncJob syncJob = iterator.next().getValue();
+                if (syncJob.isExpired(currentTimeMs)) {
+                    Map<String, List<SyncJob>> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId());

Review comment:
       使用之前判断一下key是否存在吧。




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org