You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Will-Lo (via GitHub)" <gi...@apache.org> on 2024/01/26 04:11:41 UTC

[PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Will-Lo opened a new pull request, #3868:
URL: https://github.com/apache/gobblin/pull/3868

   …al Gobblin workflows
   
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-1992
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   
   Currently the Temporal job context does not include the job metadata e.g. flow name, flow group, flow execution ID, which is critical for debugging workflows in the temporal cluster.
   
   We want to have per job logging so that it is easier to debug multiple Gobblin workflows on Temporal sent through GaaS.
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   Example logs can have this format (given the correct log4j property)
   ```
   2024-01-25 20:02:40 PST WARN  [flow.group:iceberg-persist-wus-EmailFeedbackEvent flow.name:iceberg-persist-wus-EmailFeedbackEvent flow.executionId:iceberg-persist-wus-EmailFeedbackEvent] [TaskExecutor-0] org.apache.gobblin.runtime.Task  - Synchronous task execution model is deprecated. Please consider using stream model.
   2024-01-25 20:02:40 PST INFO  [flow.group:iceberg-persist-wus-EmailFeedbackEvent flow.name:iceberg-persist-wus-EmailFeedbackEvent flow.executionId:iceberg-persist-wus-EmailFeedbackEvent] [Activity Executor taskQueue="GobblinTemporalDDM2PoCTaskQueue", namespace="gobblin-fastingest-internpoc": 1] org.apache.gobblin.runtime.GobblinMultiTaskAttempt  - Waiting for submitted tasks of job job_kip-test_iceberg-persist-wus-EmailFeedbackEvent_IcebergBasedDistcpToHdfs_holdem-oh-iceberg_war-oh-iceberg_inter-cluster-oh-iceberg-based-copy_0_1705956881307 to complete in container container-id-for-wu-0...
   2024-01-25 20:02:40 PST INFO  [flow.group:iceberg-persist-wus-EmailFeedbackEvent flow.name:iceberg-persist-wus-EmailFeedbackEvent flow.executionId:iceberg-persist-wus-EmailFeedbackEvent] [Activity Executor taskQueue="GobblinTemporalDDM2PoCTaskQueue", namespace="gobblin-fastingest-internpoc": 1] org.apache.gobblin.runtime.GobblinMultiTaskAttempt  - 1 out of 1 tasks of job job_kip-test_iceberg-persist-wus-EmailFeedbackEvent_IcebergBasedDistcpToHdfs_holdem-oh-iceberg_war-oh-iceberg_inter-cluster-oh-iceberg-based-copy_0_1705956881307 are running in container container-id-for-wu-0. 0 tasks finished.
   2024-01-25 20:02:40 PST INFO  [flow.group:iceberg-persist-wus-EmailFeedbackEvent flow.name:iceberg-persist-wus-EmailFeedbackEvent flow.executionId:iceberg-persist-wus-EmailFeedbackEvent] [TaskExecutor-0] org.apache.gobblin.runtime.TaskExecutor  - Submitting fork 0 of task task_kip-test_iceberg-persist-wus-EmailFeedbackEvent_IcebergBasedDistcpToHdfs_holdem-oh-iceberg_war-oh-iceberg_inter-cluster-oh-iceberg-based-copy_0_1705956881307_0
   2024-01-25 20:02:40 PST INFO  [flow.group:iceberg-persist-wus-EmailFeedbackEvent flow.name:iceberg-persist-wus-EmailFeedbackEvent flow.executionId:iceberg-persist-wus-EmailFeedbackEvent] [TaskExecutor-0] org.apache.gobblin.runtime.Task  - Task mode streaming = false
   ```
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#discussion_r1470135400


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs

Review Comment:
   would it be worth adding that link to the temporal.io docs into the javadoc itself?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo merged PR #3868:
URL: https://github.com/apache/gobblin/pull/3868


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#discussion_r1468234972


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs

Review Comment:
   https://api-docs.temporal.io/#temporal.api.common.v1.Header
   They're part of metadata that gets passed between workflows and activities. The ContextPropagators and Interceptors can read from these headers. When you search up workflows in the server side UI you can inspect this metadata (although they're serialized)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "phet (via GitHub)" <gi...@apache.org>.
phet commented on code in PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#discussion_r1468017947


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
       throw new RuntimeException(e);
     }
   }
+
+  private void setupMDCFromJobState(JobState jobState) {

Review Comment:
   I expect this will be reused for other job launchers, even if we're merely creating them during dev (like the GenWUs I'm about to release).  to support that, maybe situate in `Help` or `JobStateUtils`?
   
   and since I believe this MDC stuff all happens transparently behind the scenes, maybe less impl jargon in the name.  e.g. `propagateGaaSFlowExecutionContext` or `initFlowExecutionContext`?



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
       throw new RuntimeException(e);
     }
   }
+
+  private void setupMDCFromJobState(JobState jobState) {
+    // TODO: log4j2 has better syntax around conditional logging such that the key does not need to be included in the value
+    MDC.put(ConfigurationKeys.FLOW_NAME_KEY, String.format("%s:%s",ConfigurationKeys.FLOW_NAME_KEY, jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY, "NA")));
+    MDC.put(ConfigurationKeys.FLOW_GROUP_KEY, String.format("%s:%s",ConfigurationKeys.FLOW_GROUP_KEY, jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY, "NA")));

Review Comment:
   nit: I don't like `NA`, since it's technically possible that could be a valid gaas flow group.  would this token work - `<<NOT SET>>`?



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs
+ */
+@Slf4j
+public class MDCContextPropagator implements ContextPropagator {
+
+  public String getName() {
+    return this.getClass().getName();
+  }
+
+  public Object getCurrentContext() {

Review Comment:
   nit: I presume most, if not all of these methods implement declarations from `ContextPropagator`.  let's make clear which do by adding `@Override` to them



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
       throw new RuntimeException(e);
     }
   }
+
+  private void setupMDCFromJobState(JobState jobState) {
+    // TODO: log4j2 has better syntax around conditional logging such that the key does not need to be included in the value

Review Comment:
   the key may actually assist machine processing.  I'd suggest just changing the key names, so they're actually ready to be fed directly back into the GaaS REST endpoints, such as:
   ```
   GET /flowexecutions/(flowGroup:<groupName>,flowName:<flowName>,flowExecutionId:<flowExecutionId>)
   ```
   hence `flowName`, rather than `flow.name`.  bonus points if these can come out comma-separated, and so ready to use w/ that REST API, rather than space-separated as you showed in the example output.



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/work/assistance/MDCContextPropagator.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.work.assistance;
+
+import com.google.protobuf.ByteString;
+import io.temporal.api.common.v1.Payload;
+import io.temporal.common.context.ContextPropagator;
+import java.nio.charset.StandardCharsets;
+import java.util.HashMap;
+import java.util.Map;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.MDC;
+
+
+/**
+ * Context propagator for MDC in Temporal GaaS flows, which is used to share context between workflows and activities.
+ * Keys added to MDC will show up in Temporal headers serialized as protobufs

Review Comment:
   not clear what "Temporal headers" are... is this about HTTP requests?  please explain



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "Will-Lo (via GitHub)" <gi...@apache.org>.
Will-Lo commented on code in PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#discussion_r1468236220


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ProcessWorkUnitsJobLauncher.java:
##########
@@ -93,4 +97,11 @@ public void submitJob(List<WorkUnit> workunits) {
       throw new RuntimeException(e);
     }
   }
+
+  private void setupMDCFromJobState(JobState jobState) {
+    // TODO: log4j2 has better syntax around conditional logging such that the key does not need to be included in the value

Review Comment:
   Yeah this is a good point, it might make it harder to set up queries on logs though if we keep the keys in the values. I'll change the keys to match what the API accepts and format the logs appropriately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [GOBBLIN-1992] Propagate slf4j MDC Context in workflows and activities of the tempor… [gobblin]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3868:
URL: https://github.com/apache/gobblin/pull/3868#issuecomment-1911468345

   ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3868?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`318c661`)](https://app.codecov.io/gh/apache/gobblin/commit/318c661808d201f72090088aa975f39a06f375eb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 47.61% compared to head [(`e9a8ffd`)](https://app.codecov.io/gh/apache/gobblin/pull/3868?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 46.00%.
   > Report is 3 commits behind head on master.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3868      +/-   ##
   ============================================
   - Coverage     47.61%   46.00%   -1.61%     
   + Complexity    11078     2187    -8891     
   ============================================
     Files          2160      417    -1743     
     Lines         85511    18054   -67457     
     Branches       9478     2199    -7279     
   ============================================
   - Hits          40713     8306   -32407     
   + Misses        41111     8864   -32247     
   + Partials       3687      884    -2803     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3868?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org