[GitHub] [incubator-gobblin] ibuenros commented on a change in pull request #2636: [GOBBLIN-766] Emit Workunits Created event

[GitBox <gi...@apache.org>]

ibuenros commented on a change in pull request #2636: [GOBBLIN-766]  Emit Workunits Created event
URL: https://github.com/apache/incubator-gobblin/pull/2636#discussion_r288294736
 
 

 ##########
 File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java
 ##########
 @@ -377,6 +378,13 @@ public void apply(JobListener jobListener, JobContext jobContext)
           jobState.setState(JobState.RunningState.COMMITTED);
           isWorkUnitsEmpty = true;
           return;
+        } else {
+          if (workUnitStream.isSafeToMaterialize()) {
+            this.eventSubmitter.submit(JobEvent.WORK_UNITS_CREATED, "workUnitsCreated",
+                Long.toString(workUnitStream.getMaterializedWorkUnitCollection().size()));
 
 Review comment:
   This call will force work units to be materialized and prevent all optimizations from streaming them (in terms of memory and speed). Ideally, the correct way of doing this is to count the work units as we work on them and emit the event at the end of the run.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services