You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/08/11 23:04:09 UTC
[GitHub] [gobblin] aplex commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r687241795
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
Review comment:
FLOW_GROUP / FLOW_NAME can be null when the Gobblin job is executed outside of Gobblin service (just as Azkaban flow). This also means that "ServiceMetricNames.GOBBLIN_SERVICE_PREFIX" would not be applicable to such jobs.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
Review comment:
The base class has optional runtimeMetricContext. Can we use that? I think it will add some extra tags to metrics to tell them apart across the job.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge guage = metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
Where do we emit other metrics for GaaS-triggered jobs - inside the job itself or in the GaaS code?
I see that we are emitting CountEvent for work unit size earlier in the code. I assume it will land in GaaS and be available for processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org