[GitHub] [gobblin] phet commented on a change in pull request #3399: Add retries to KafkaJobStatusMonitor for recovery from transient JobStatus state store failure.

[GitBox <gi...@apache.org>]

phet commented on a change in pull request #3399:
URL: https://github.com/apache/gobblin/pull/3399#discussion_r713689693



##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaJobStatusMonitor.java
##########
@@ -90,6 +100,10 @@
   @Getter
   private final StateStore<org.apache.gobblin.configuration.State> stateStore;
   private final ScheduledExecutorService scheduledExecutorService;
+  private static final Config RETRYER_FALLBACK_CONFIG = ConfigFactory.parseMap(ImmutableMap.of(
+      RETRY_TIME_OUT_MS, TimeUnit.HOURS.toMillis(24L), // after a day, presume non-transient and give up

Review comment:
       we could consider indefinite retries, although I wanted to set a finite boundary to avoid "infinite looping".  if the error we're presuming transient takes >24 hours to resolve it probably merits human intervention.  at that point an operator is there to restart the service.  realistically, the probability of a truly transient error continuing >24 hours seems low.
   
   if actually a problem with the message itself and/or incompatibility against the DB schema, that may require manual advance of the offset to skip the message or to adjust systems.  concern we might misdiagnose something like that as a transient error made me want to prevent an infinite loop.
   
   still, while this is finite, it may always be overridden by config:
   ```
     public static final String KAFKA_JOB_STATUS_MONITOR_RETRY_TIME_OUT_MINUTES =
         "gobblin.kafka.jobStatusMonitor.retry.timeOut.minutes";
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org