You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/11/09 08:53:00 UTC

[GitHub] [incubator-doris] xy720 opened a new pull request #7061: [Feature] Clean up old sync jobs regularly

xy720 opened a new pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061


   ## Proposed changes
   
   #7060 
   
   Each job that has been stopped for more than 3 days(set with Config.label_keep_max_second) will be permanently cleaned up.
   
   ## Types of changes
   
   What types of changes does your code introduce to Doris?
   _Put an `x` in the boxes that apply_
   
   - [x] Bugfix (non-breaking change which fixes an issue)
   - [x] New feature (non-breaking change which adds functionality)
   
   ## Checklist
   
   _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
   
   - [x] I have created an issue on (Fix #7060 ) and described the bug/feature there in detail
   - [x] Compiling and unit tests pass locally with my changes
   - [x] I have added tests that prove my fix is effective or that my feature works
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] pengxiangyu commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly

Posted by GitBox <gi...@apache.org>.
pengxiangyu commented on a change in pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r745579688



##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java
##########
@@ -73,5 +73,7 @@ private void process() throws UserException {
                 job.cancel(msgType, exception.getMessage());
             }
         }
+
+        this.syncJobManager.cleanOldSyncJobs();

Review comment:
       cleanOldSyncJobs() 放在runAfterCatalogReady()里更好些，新建与删除最好分开在两个函数。

##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java
##########
@@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException {
         }
     }
 
+    // Remove old sync jobs. Called periodically.
+    // Stopped jobs will be removed after Config.label_keep_max_second.
+    public void cleanOldSyncJobs() {
+        LOG.debug("begin to clean old sync jobs ");
+        long currentTimeMs = System.currentTimeMillis();
+        writeLock();
+        try {
+            Iterator<Map.Entry<Long, SyncJob>> iterator = idToSyncJob.entrySet().iterator();
+            while (iterator.hasNext()) {
+                SyncJob syncJob = iterator.next().getValue();
+                if (syncJob.isExpired(currentTimeMs)) {
+                    Map<String, List<SyncJob>> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId());

Review comment:
       使用之前判断一下key是否存在吧。




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7061: [Feature] Clean up old sync jobs regularly

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#issuecomment-966385024


   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] xy720 commented on a change in pull request #7061: [Feature] Clean up old sync jobs regularly

Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#discussion_r746206761



##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncChecker.java
##########
@@ -73,5 +73,7 @@ private void process() throws UserException {
                 job.cancel(msgType, exception.getMessage());
             }
         }
+
+        this.syncJobManager.cleanOldSyncJobs();

Review comment:
       ok. done

##########
File path: fe/fe-core/src/main/java/org/apache/doris/load/sync/SyncJobManager.java
##########
@@ -279,6 +280,40 @@ public void updateNeedSchedule() throws UserException {
         }
     }
 
+    // Remove old sync jobs. Called periodically.
+    // Stopped jobs will be removed after Config.label_keep_max_second.
+    public void cleanOldSyncJobs() {
+        LOG.debug("begin to clean old sync jobs ");
+        long currentTimeMs = System.currentTimeMillis();
+        writeLock();
+        try {
+            Iterator<Map.Entry<Long, SyncJob>> iterator = idToSyncJob.entrySet().iterator();
+            while (iterator.hasNext()) {
+                SyncJob syncJob = iterator.next().getValue();
+                if (syncJob.isExpired(currentTimeMs)) {
+                    Map<String, List<SyncJob>> map = dbIdToJobNameToSyncJobs.get(syncJob.getDbId());

Review comment:
       ok




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] github-actions[bot] commented on pull request #7061: [Feature] Clean up old sync jobs regularly

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061#issuecomment-964736879


   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] morningman merged pull request #7061: [Feature] Clean up old sync jobs regularly

Posted by GitBox <gi...@apache.org>.
morningman merged pull request #7061:
URL: https://github.com/apache/incubator-doris/pull/7061


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org