[GitHub] [gobblin] phet commented on a change in pull request #3357: [GOBBLIN-1509] Announce flow failure on DagManager::addDag error

[GitBox <gi...@apache.org>]

phet commented on a change in pull request #3357:
URL: https://github.com/apache/gobblin/pull/3357#discussion_r685762496



##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java
##########
@@ -288,13 +287,28 @@ public void orchestrate(Spec spec) throws Exception {
       flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
           jobExecutionPlanDag.getNodes().get(0).getValue().getJobSpec().getConfigAsProperties().getProperty(ConfigurationKeys.FLOW_EXECUTION_ID_KEY));
 
-      if (flowCompilationTimer != null) {
-        flowCompilationTimer.stop(flowMetadata);
+      if (flowCompilationTimer.isPresent()) {
+        flowCompilationTimer.get().stop(flowMetadata);
       }
 
       if (this.dagManager.isPresent()) {
-        //Send the dag to the DagManager.
-        this.dagManager.get().addDag(jobExecutionPlanDag, true, true);
+        try {
+          //Send the dag to the DagManager.
+          this.dagManager.get().addDag(jobExecutionPlanDag, true, true);
+        } catch (Exception ex) {
+          // pronounce failed before stack unwinds, to ensure flow not marooned in `COMPILED` state; (failure likely attributable to DB connection/failover)
+          String context = String.format("flow: %s; exec: %s",
+              DagManagerUtils.getFlowId(jobExecutionPlanDag).toString(),
+              DagManagerUtils.getFlowExecId(jobExecutionPlanDag));
+          String failureMessage = "Failed to add Job Execution Plan due to: " + ex.getMessage();
+          _log.warn(String.format("[%s] %s", context, failureMessage));

Review comment:
       I didn't want to double up on the stack trace, since I re-throw just after.  the final handler can log the trace.  I give only the exception's message for correlation.
   
   in the thread below, we could discuss whether to log here at all.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org