[GitHub] [incubator-doris] xy720 commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly

[GitBox <gi...@apache.org>]

xy720 commented on a change in pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r746206761



##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java
##########
@@ -73,5 +73,7 @@ private void process() throws UserException {
                 job.cancel(msgType, exception.getMessage());
             }
         }
+
+        this.syncJobManager.cleanOldSyncJobs();

Review comment:
       ok. done

##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java
##########
@@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException {
         }
     }
 
+    // Remove old sync jobs. Called periodically.
+    // Stopped jobs will be removed after Config.label_keep_max_second.
+    public void cleanOldSyncJobs() {
+        LOG.debug("begin to clean old sync jobs ");
+        long currentTimeMs = System.currentTimeMillis();
+        writeLock();
+        try {
+            Iterator<Map.Entry<Long, SyncJob>> iterator = idToSyncJob.entrySet().iterator();
+            while (iterator.hasNext()) {
+                SyncJob syncJob = iterator.next().getValue();
+                if (syncJob.isExpired(currentTimeMs)) {
+                    Map<String, List<SyncJob>> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId());

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org