You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/04/02 22:43:57 UTC

[GitHub] [incubator-pinot] xiaohui-sun opened a new pull request #5208: [TE] Disable alerts if it has no success run within 30 days

xiaohui-sun opened a new pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208
 
 
   Check alert's lastTaskRunTime. If there is no success run within 30 days then disable it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407146946
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
 
 Review comment:
   Currently we don't have a good way to test it. The only way would be integration test. But having more than one integration tests running may cause conflicts. That's why I have to add sequence number in our local integration test.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405814064
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
 
 Review comment:
   nit: typo `currentTimeMillis` & `maxTaskFailMillis`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407145114
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
+    long maxTaskFailMills = TimeUnit.DAYS.toMillis(MAX_FAILED_DISABLE_DAYS);
+    for (DetectionConfigDTO config : detectionConfigs) {
+      try {
+        Timestamp updateTime = config.getUpdateTime();
+        if (updateTime != null && config.getHealth() != null && config.getHealth().getDetectionTaskStatus() != null) {
+          long lastTaskExecutionTime = config.getHealth().getDetectionTaskStatus().getLastTaskExecutionTime();
+          if (updateTime.getTime() <= currentTimeMills - maxTaskFailMills && (lastTaskExecutionTime == -1L
 
 Review comment:
   lastTaskExecutionTime == -1L is used for backward compatibility.
   Currently we have many long failing alerts have -1L.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407146873
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
+    long maxTaskFailMills = TimeUnit.DAYS.toMillis(MAX_FAILED_DISABLE_DAYS);
+    for (DetectionConfigDTO config : detectionConfigs) {
+      try {
+        Timestamp updateTime = config.getUpdateTime();
+        if (updateTime != null && config.getHealth() != null && config.getHealth().getDetectionTaskStatus() != null) {
+          long lastTaskExecutionTime = config.getHealth().getDetectionTaskStatus().getLastTaskExecutionTime();
+          if (updateTime.getTime() <= currentTimeMills - maxTaskFailMills && (lastTaskExecutionTime == -1L
+              || lastTaskExecutionTime <= currentTimeMills - maxTaskFailMills)) {
+            config.setActive(false);
+            detectionDAO.update(config);
+            sendDisableAlertNotificationEmail(config);
+            LOG.info("Disabled alert " + config.getId() + " since it failed more than " + MAX_FAILED_DISABLE_DAYS + " days");
+            LOG.info("Task last update time: " + config.getUpdateTime());
+            LOG.info("Last success task execution time: " + lastTaskExecutionTime);
+          }
+        }
+      } catch (Exception e) {
+        LOG.error("Exception in disabling alert ", config.getId());
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407146852
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
 
 Review comment:
   fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405815219
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
+    long maxTaskFailMills = TimeUnit.DAYS.toMillis(MAX_FAILED_DISABLE_DAYS);
+    for (DetectionConfigDTO config : detectionConfigs) {
+      try {
+        Timestamp updateTime = config.getUpdateTime();
+        if (updateTime != null && config.getHealth() != null && config.getHealth().getDetectionTaskStatus() != null) {
+          long lastTaskExecutionTime = config.getHealth().getDetectionTaskStatus().getLastTaskExecutionTime();
+          if (updateTime.getTime() <= currentTimeMills - maxTaskFailMills && (lastTaskExecutionTime == -1L
 
 Review comment:
   Can we skip the `lastTaskExecutionTime == -1L` check?
   Isn't that already covered by `lastTaskExecutionTime <= currentTimeMills - maxTaskFailMills`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405803905
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
+    long maxTaskFailMills = TimeUnit.DAYS.toMillis(MAX_FAILED_DISABLE_DAYS);
+    for (DetectionConfigDTO config : detectionConfigs) {
+      try {
+        Timestamp updateTime = config.getUpdateTime();
+        if (updateTime != null && config.getHealth() != null && config.getHealth().getDetectionTaskStatus() != null) {
+          long lastTaskExecutionTime = config.getHealth().getDetectionTaskStatus().getLastTaskExecutionTime();
+          if (updateTime.getTime() <= currentTimeMills - maxTaskFailMills && (lastTaskExecutionTime == -1L
+              || lastTaskExecutionTime <= currentTimeMills - maxTaskFailMills)) {
+            config.setActive(false);
+            detectionDAO.update(config);
+            sendDisableAlertNotificationEmail(config);
+            LOG.info("Disabled alert " + config.getId() + " since it failed more than " + MAX_FAILED_DISABLE_DAYS + " days");
+            LOG.info("Task last update time: " + config.getUpdateTime());
+            LOG.info("Last success task execution time: " + lastTaskExecutionTime);
+          }
+        }
+      } catch (Exception e) {
+        LOG.error("Exception in disabling alert ", config.getId());
 
 Review comment:
   log the exception.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun merged pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun merged pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405816169
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
+    DetectionConfigManager detectionDAO = DAO_REGISTRY.getDetectionConfigManager();
+    List<DetectionConfigDTO> detectionConfigs = detectionDAO.findAllActive();
+    long currentTimeMills = System.currentTimeMillis();
+    long maxTaskFailMills = TimeUnit.DAYS.toMillis(MAX_FAILED_DISABLE_DAYS);
+    for (DetectionConfigDTO config : detectionConfigs) {
+      try {
+        Timestamp updateTime = config.getUpdateTime();
+        if (updateTime != null && config.getHealth() != null && config.getHealth().getDetectionTaskStatus() != null) {
+          long lastTaskExecutionTime = config.getHealth().getDetectionTaskStatus().getLastTaskExecutionTime();
+          if (updateTime.getTime() <= currentTimeMills - maxTaskFailMills && (lastTaskExecutionTime == -1L
+              || lastTaskExecutionTime <= currentTimeMills - maxTaskFailMills)) {
+            config.setActive(false);
+            detectionDAO.update(config);
+            sendDisableAlertNotificationEmail(config);
+            LOG.info("Disabled alert " + config.getId() + " since it failed more than " + MAX_FAILED_DISABLE_DAYS + " days");
+            LOG.info("Task last update time: " + config.getUpdateTime());
+            LOG.info("Last success task execution time: " + lastTaskExecutionTime);
 
 Review comment:
   Clubbing them into one log statement might be easier to debug.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r407146245
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
 
 Review comment:
   Yes we want to handle it more conservatively. Give them 30 days grace period.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405811811
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
+   */
+  private void disableLongFailedAlerts() {
 
 Review comment:
   Can we include some unit tests for this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days

Posted by GitBox <gi...@apache.org>.
akshayrai commented on a change in pull request #5208: [TE] Disable alerts if it has no success run within 30 days
URL: https://github.com/apache/incubator-pinot/pull/5208#discussion_r405818464
 
 

 ##########
 File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
 ##########
 @@ -120,11 +134,62 @@ private void executeMonitorUpdate(MonitorTaskInfo monitorTaskInfo) {
 
       // update detection health
       updateDetectionHealth();
+
+      // disable alerts that failed consecutively for a long time
+      disableLongFailedAlerts();
+
     } catch (Exception e) {
       LOG.error("Exception in monitor update task", e);
     }
   }
 
+  /**
+   * Disable the alert if it was updated before {@MAX_TASK_FAIL_DAYS} but there is no success run since then.
 
 Review comment:
   What if the alert was updated(after receiving the below notification) and it still continues to fail? We don't want to disable & notify them again?
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org