You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/08/01 00:03:00 UTC

[jira] [Work logged] (GOBBLIN-1863) Multi-Active Launch Job Related Issues

     [ https://issues.apache.org/jira/browse/GOBBLIN-1863?focusedWorklogId=873917&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-873917 ]

ASF GitHub Bot logged work on GOBBLIN-1863:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Aug/23 00:02
            Start Date: 01/Aug/23 00:02
    Worklog Time Spent: 10m 
      Work Description: umustafi commented on code in PR #3727:
URL: https://github.com/apache/gobblin/pull/3727#discussion_r1279990300


##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -374,9 +358,41 @@ public void orchestrate(Spec spec, Properties jobProps, long triggerTimestampMil
     Instrumented.updateTimer(this.flowOrchestrationTimer, System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
   }
 
+  /**
+   * Abstraction used to populate the message of and emit a FlowCompileFailed event for the Orchestrator.
+   * @param spec
+   * @param flowMetadata
+   */
+  public void emitFlowCompilationFailedEvent(Spec spec, Map<String, String> flowMetadata) {
+    // For scheduled flows, we do not insert the flowExecutionId into the FlowSpec. As a result, if the flow
+    // compilation fails (i.e. we are unable to find a path), the metadata will not have flowExecutionId.
+    // In this case, the current time is used as the flow executionId.
+    flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
+        Long.toString(System.currentTimeMillis()));
+
+    String message = "Flow was not compiled successfully.";
+    if (!((FlowSpec) spec).getCompilationErrors().isEmpty()) {
+      message = message + " Compilation errors encountered: " + ((FlowSpec) spec).getCompilationErrors();
+    }
+    _log.warn(message);
+    flowMetadata.put(TimingEvent.METADATA_MESSAGE, message);
+
+    Optional<TimingEvent> flowCompileFailedTimer = eventSubmitter.transform(submitter ->
+        new TimingEvent(submitter, TimingEvent.FlowTimings.FLOW_COMPILE_FAILED));
+
+    if (flowCompileFailedTimer.isPresent()) {
+      flowCompileFailedTimer.get().stop(flowMetadata);
+    }
+  }
+
   public void submitFlowToDagManager(FlowSpec flowSpec)
       throws IOException {
-    submitFlowToDagManager(flowSpec, specCompiler.compileFlow(flowSpec));
+    Dag<JobExecutionPlan> jobExecutionPlanDag = specCompiler.compileFlow(flowSpec);
+    if (jobExecutionPlanDag == null || jobExecutionPlanDag.isEmpty()) {
+      emitFlowCompilationFailedEvent(flowSpec, TimingEventUtils.getFlowMetadata(flowSpec));
+      return;
+    }
+    submitFlowToDagManager(flowSpec, jobExecutionPlanDag);

Review Comment:
   We don't call orchestrate function directly from the `DagActionStoreMonitor` because the chain of events is `Scheduler -> Orchestrator -> FlowTriggerHandler (Multi-Active algorithm) -> MySQL -> Brooklin change stream -> DagActionStoreMonitor -> DagManager` so we don't want the MA algorithm triggered again. However, I did end up abstracting all the checks done to compile the flow and ensure an execution is allowed to re-use when submitting the flow to the `DagManager` from the `DagActionStoreMonitor`





Issue Time Tracking
-------------------

    Worklog Id:     (was: 873917)
    Time Spent: 2h  (was: 1h 50m)

> Multi-Active Launch Job Related Issues
> --------------------------------------
>
>                 Key: GOBBLIN-1863
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1863
>             Project: Apache Gobblin
>          Issue Type: Bug
>            Reporter: Urmi Mustafi
>            Priority: Major
>          Time Spent: 2h
>  Remaining Estimate: 0h
>
> * DagManager check leader status before addDag bc calling this method from non-leader hosts throws a NPE which may caused failed dag event to be emitted
>  * also handle LAUNCH type events upon leader change and setting a new participant DagManager to be active. Failing to handle these events may be causing missed flow launches on any leader change or restart. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)