[jira] [Work logged] (GOBBLIN-1750) Emit GaaSObservabilityEvent to mark job summaries from gaas

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GOBBLIN-1750?focusedWorklogId=836673&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-836673 ]

ASF GitHub Bot logged work on GOBBLIN-1750:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 03/Jan/23 17:13
            Start Date: 03/Jan/23 17:13
    Worklog Time Spent: 10m 
      Work Description: Will-Lo commented on code in PR #3610:
URL: https://github.com/apache/gobblin/pull/3610#discussion_r1060789952


##########
gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GaaSObservabilityEventExperimental.avsc:
##########
@@ -0,0 +1,167 @@
+{
+  "type": "record",
+  "name": "GaaSObservabilityEventExperimental",
+  "namespace": "org.apache.gobblin.metrics",
+  "doc": "An experimental format for GaaS to emit events during and after a job is executed.",
+  "fields": [
+    {
+      "name": "timestamp",
+      "type": "long",
+      "doc": "Time at which event was created in millis"
+    },
+    {
+      "name": "flowGroup",
+      "type": "string",
+      "doc": "Flow group for the GaaS flow",
+      "compliance": "NONE"
+    },
+    {
+      "name": "flowName",
+      "type": "string",
+      "doc": "Flow name for the GaaS flow",
+      "compliance": "NONE"
+    },
+    {
+      "name": "flowExecutionId",
+      "type": "long",
+      "doc": "Flow execution id for the GaaS flow",
+      "compliance": "NONE"
+    },
+    {
+      "name": "lastFlowModificationTime",
+      "type": "long",
+      "doc": "Timestamp in millis since Epoch when the flow config was last modified"
+    },
+    {
+      "name": "flowGraphEdgeId",
+      "type": "string",
+      "doc": "Flow edge id, in format <src_node>_<dest_node>_<edge_id>",
+      "compliance": "NONE"
+    },
+    {
+      "name": "jobName",
+      "type": "string",
+      "doc": "The name of the Gobblin job, found in the job template. One edge can contain multiple jobs",
+      "compliance": "NONE"
+    },
+    {
+      "name": "jobStatus",
+      "type": {
+        "type": "enum",
+        "name": "JobStatus",
+        "symbols": [
+          "SUCCEEDED",
+          "COMPILATION_FAILURE",
+          "SUBMISSION_FAILURE",
+          "EXEUCTION_FAILURE",
+          "CANCELLED"
+        ],
+        "doc": "Final job status for this job in the GaaS flow",
+        "compliance": "NONE"
+      }
+    },
+    {
+      "name": "jobOrchestratedTime",
+      "type": [
+        "null",
+        "long"
+      ],
+      "doc": "Timestamp when the job was successfully sent to the job executor, null if it was unable to be sent."
+    },
+    {
+      "name": "jobStartTime",
+      "type": "long",
+      "doc": "Start time of the job in millis since Epoch",
+      "compliance": "NONE"
+    },
+    {
+      "name": "jobEndTime",
+      "type": "long",
+      "doc": "Finish time of the job in millis since Epoch",
+      "compliance": "NONE"
+    },
+    {
+      "name": "executionUserUrn",
+      "type": [
+        "null",
+        "string"
+      ],
+      "doc": "User URN (if applicable) that runs the underlying Gobblin job",
+      "compliance": "NONE"
+    },
+    {
+      "name": "executorUrl",
+      "type": [
+        "null",

Review Comment:
   At least according to the LinkedIn DMRC they much prefer null defaults and values over non-null defaults. Null types and defaults makes the contracts more explicit





Issue Time Tracking
-------------------

    Worklog Id:     (was: 836673)
    Time Spent: 4h 50m  (was: 4h 40m)

> Emit GaaSObservabilityEvent to mark job summaries from gaas
> -----------------------------------------------------------
>
>                 Key: GOBBLIN-1750
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1750
>             Project: Apache Gobblin
>          Issue Type: New Feature
>          Components: gobblin-service
>            Reporter: William Lo
>            Assignee: Abhishek Tiwari
>            Priority: Major
>          Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> GobblinTrackingEvents work for service <-> service communication, but it is hard to set up alerts or write queries around them in the GaaS context, due to lacking all of the context and having varying amounts of details around the Gobblin job.
> We want to emit a single GaaS event for jobs running on GaaS that provides a summary and can be easily queryable/parseable for other systems and engines to quickly understand the status of a Gobblin job.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)