[GitHub] [gobblin] jack-moseley commented on a change in pull request #3220: [GOBBLIN-1380] Add retention to failed dag state store

[GitBox <gi...@apache.org>]

jack-moseley commented on a change in pull request #3220:
URL: https://github.com/apache/gobblin/pull/3220#discussion_r572405809



##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
     }
   }
 
+  /**
+   * Thread that runs retention on failed dags based on their original start time (from flow execution ID).
+   */
+  public static class FailedDagRetentionThread implements Runnable {
+    private final DagStateStore failedDagStateStore;
+    private final Map<String, Dag<JobExecutionPlan>> failedDags;
+    private final long failedDagRetentionTime;
+
+    FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String, Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+      this.failedDagStateStore = failedDagStateStore;
+      this.failedDags = failedDags;
+      this.failedDagRetentionTime = failedDagRetentionTime;
+    }
+
+    @Override
+    public void run() {
+      try {
+        log.info("start clean");
+        for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter = this.failedDags.entrySet().iterator(); iter.hasNext();) {
+          Map.Entry<String, Dag<JobExecutionPlan>> entry = iter.next();
+          if (this.failedDagRetentionTime > 0L
+              && System.currentTimeMillis() > DagManagerUtils.getFlowExecId(entry.getValue()) + this.failedDagRetentionTime) {
+            log.info("cleaning");

Review comment:
       Whoops, I added those logs for testing and meant to remove them.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org