You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "abhishekmjain (via GitHub)" <gi...@apache.org> on 2025/02/04 14:20:21 UTC

[PR] [GOBBLIN-2192] Add temporal worker metrics support [gobblin]

abhishekmjain opened a new pull request, #4095:
URL: https://github.com/apache/gobblin/pull/4095

   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
   
   
   ### JIRA
   - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN-2192) issues and references them in the PR title. For example, "[GOBBLIN-2192] My Gobblin PR"
       - https://issues.apache.org/jira/browse/GOBBLIN-2192
   
   
   ### Description
   - [ ] Here are some details about my PR, including screenshots (if applicable):
   
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
   - testGetDimensions
   - testGetDimensionsEmptyConfig
   - testGetOtlpConfig
   
   Also performed local project testing
   
   ### Commits
   - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
       1. Subject is separated from body by a blank line
       2. Subject is limited to 50 characters
       3. Subject does not end with a period
       4. Subject uses the imperative mood ("add", not "adding")
       5. Body wraps at 72 characters
       6. Body explains "what" and "why", not "how"
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2192] Add temporal worker metrics support [gobblin]

Posted by "Blazer-007 (via GitHub)" <gi...@apache.org>.
Blazer-007 commented on code in PR #4095:
URL: https://github.com/apache/gobblin/pull/4095#discussion_r1943048684


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/GobblinTemporalConfigurationKeys.java:
##########
@@ -74,4 +74,14 @@ public interface GobblinTemporalConfigurationKeys {
 
   String DYNAMIC_SCALING_POLLING_INTERVAL_SECS = DYNAMIC_SCALING_PREFIX + "polling.interval.seconds";
   int DEFAULT_DYNAMIC_SCALING_POLLING_INTERVAL_SECS = 60;
+
+  /**
+   * Temporal metrics config properties
+   */
+  String TEMPORAL_METRICS_PREFIX = PREFIX + "metrics.";
+  String TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT = TEMPORAL_METRICS_PREFIX + "otlp";
+  String TEMPORAL_METRICS_OTLP_HEADERS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".headers";
+  String TEMPORAL_METRICS_REPORT_INTERVAL_SECS = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".report.interval.seconds";
+  int DEFAULT_TEMPORAL_METRICS_REPORT_INTERVAL_SECS = 10;
+  String TEMPORAL_METRICS_OTLP_DIMENSIONS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".dimensions";

Review Comment:
   What does these dimensions represent ?



##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/GobblinTemporalConfigurationKeys.java:
##########
@@ -74,4 +74,14 @@ public interface GobblinTemporalConfigurationKeys {
 
   String DYNAMIC_SCALING_POLLING_INTERVAL_SECS = DYNAMIC_SCALING_PREFIX + "polling.interval.seconds";
   int DEFAULT_DYNAMIC_SCALING_POLLING_INTERVAL_SECS = 60;
+
+  /**
+   * Temporal metrics config properties
+   */
+  String TEMPORAL_METRICS_PREFIX = PREFIX + "metrics.";
+  String TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT = TEMPORAL_METRICS_PREFIX + "otlp";
+  String TEMPORAL_METRICS_OTLP_HEADERS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".headers";
+  String TEMPORAL_METRICS_REPORT_INTERVAL_SECS = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".report.interval.seconds";
+  int DEFAULT_TEMPORAL_METRICS_REPORT_INTERVAL_SECS = 10;
+  String TEMPORAL_METRICS_OTLP_DIMENSIONS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".dimensions";

Review Comment:
   Also are these configs necessary to create i.e. cant we use open telemetry configs & their passed values ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2192] Add temporal worker metrics support [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain commented on code in PR #4095:
URL: https://github.com/apache/gobblin/pull/4095#discussion_r1943057422


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/GobblinTemporalConfigurationKeys.java:
##########
@@ -74,4 +74,14 @@ public interface GobblinTemporalConfigurationKeys {
 
   String DYNAMIC_SCALING_POLLING_INTERVAL_SECS = DYNAMIC_SCALING_PREFIX + "polling.interval.seconds";
   int DEFAULT_DYNAMIC_SCALING_POLLING_INTERVAL_SECS = 60;
+
+  /**
+   * Temporal metrics config properties
+   */
+  String TEMPORAL_METRICS_PREFIX = PREFIX + "metrics.";
+  String TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT = TEMPORAL_METRICS_PREFIX + "otlp";
+  String TEMPORAL_METRICS_OTLP_HEADERS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".headers";
+  String TEMPORAL_METRICS_REPORT_INTERVAL_SECS = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".report.interval.seconds";
+  int DEFAULT_TEMPORAL_METRICS_REPORT_INTERVAL_SECS = 10;
+  String TEMPORAL_METRICS_OTLP_DIMENSIONS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".dimensions";

Review Comment:
   The tags for the metrics. Like let's say we want to filter on flow.name, we can use that as a dimension and view metrics at individual flow.name level. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2192] Add temporal worker metrics support [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain commented on code in PR #4095:
URL: https://github.com/apache/gobblin/pull/4095#discussion_r1943070312


##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/GobblinTemporalConfigurationKeys.java:
##########
@@ -74,4 +74,14 @@ public interface GobblinTemporalConfigurationKeys {
 
   String DYNAMIC_SCALING_POLLING_INTERVAL_SECS = DYNAMIC_SCALING_PREFIX + "polling.interval.seconds";
   int DEFAULT_DYNAMIC_SCALING_POLLING_INTERVAL_SECS = 60;
+
+  /**
+   * Temporal metrics config properties
+   */
+  String TEMPORAL_METRICS_PREFIX = PREFIX + "metrics.";
+  String TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT = TEMPORAL_METRICS_PREFIX + "otlp";
+  String TEMPORAL_METRICS_OTLP_HEADERS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".headers";
+  String TEMPORAL_METRICS_REPORT_INTERVAL_SECS = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".report.interval.seconds";
+  int DEFAULT_TEMPORAL_METRICS_REPORT_INTERVAL_SECS = 10;
+  String TEMPORAL_METRICS_OTLP_DIMENSIONS_KEY = TEMPORAL_METRICS_OTLP_PREFIX_WITHOUT_DOT + ".dimensions";

Review Comment:
   We need to parse headers separately since the default config of header parsing considers ',' as split value. There may be a header key or value which needs ',' in its string which will not work if we don't have an object mapper.
   For dimension there is no config, have mentioned the benefits in above comment.
   For step, there's a default value of 1 min, which seems a bit longer than the usual 10 seconds reporting interval.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GOBBLIN-2192] Add temporal worker metrics support [gobblin]

Posted by "abhishekmjain (via GitHub)" <gi...@apache.org>.
abhishekmjain merged PR #4095:
URL: https://github.com/apache/gobblin/pull/4095


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org