You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2021/08/04 22:56:49 UTC
[GitHub] [gobblin] umustafi opened a new pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
umustafi opened a new pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351
*Emitted in the form <flowgroup>.<flowname>.workunits
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [X] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1505
### Description
- [X] Here are some details about my PR, including screenshots (if applicable):
From a user/platform perspective, we want to see how many workunits are being created for each flow. The metric of workunit count will be emitted in the form <flowgroup>.<flowname>.workunits.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
Tested with flows launched on gaas to see log statement "Reporting metrics for host: {}, app: {} to Ingraphs with reporting interval of {} seconds."
### Commits
- [X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f8aaa7b) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
=========================================
Coverage 46.42% 46.43%
+ Complexity 10115 10113 -2
=========================================
Files 2052 2052
Lines 79626 79634 +8
Branches 8889 8889
=========================================
+ Hits 36970 36976 +6
+ Misses 39216 39215 -1
- Partials 3440 3443 +3
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.73% <100.00%> (+0.35%)` | :arrow_up: |
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.01% <100.00%> (+0.45%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0.00%> (+1.42%)` | :arrow_up: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.71% <0.00%> (+2.47%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...f8aaa7b](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r696177667
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -64,6 +72,8 @@
@Getter
private Meter messageParseFailures;
+ private final HashMap<String, Long> flowNameGroupToWorkUnitCount;
Review comment:
used a ConcurrentHashMap to make it thread-safe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a14fdb2) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `0.01%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 46.41% -0.02%
- Complexity 10120 10155 +35
============================================
Files 2052 2056 +4
Lines 79656 80102 +446
Branches 8896 8950 +54
============================================
+ Hits 36991 37183 +192
- Misses 39226 39472 +246
- Partials 3439 3447 +8
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [...blin/service/monitoring/KafkaJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUpvYlN0YXR1c01vbml0b3IuamF2YQ==) | `35.57% <0.00%> (-7.45%)` | :arrow_down: |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `27.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `32.23% <0.00%> (-2.48%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| ... and [55 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...a14fdb2](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (63ede5f) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `4.01%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 42.41% -4.02%
+ Complexity 10120 4543 -5577
============================================
Files 2052 1025 -1027
Lines 79656 40663 -38993
Branches 8896 4537 -4359
============================================
- Hits 36991 17249 -19742
+ Misses 39226 21702 -17524
+ Partials 3439 1712 -1727
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...g/apache/gobblin/hive/HiveMetastoreClientPool.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVNZXRhc3RvcmVDbGllbnRQb29sLmphdmE=) | | |
| [...rg/apache/gobblin/writer/WatermarkAwareWriter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL1dhdGVybWFya0F3YXJlV3JpdGVyLmphdmE=) | | |
| [...che/gobblin/writer/HiveWritableHdfsDataWriter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9IaXZlV3JpdGFibGVIZGZzRGF0YVdyaXRlci5qYXZh) | | |
| [.../gobblin/compaction/verify/CompactionVerifier.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vdmVyaWZ5L0NvbXBhY3Rpb25WZXJpZmllci5qYXZh) | | |
| [...agement/version/TimestampedHiveDatasetVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWRIaXZlRGF0YXNldFZlcnNpb24uamF2YQ==) | | |
| [...che/gobblin/util/ReflectivePredicateEvaluator.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvUmVmbGVjdGl2ZVByZWRpY2F0ZUV2YWx1YXRvci5qYXZh) | | |
| [...nt/copy/UnixTimestampRecursiveCopyableDataset.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVW5peFRpbWVzdGFtcFJlY3Vyc2l2ZUNvcHlhYmxlRGF0YXNldC5qYXZh) | | |
| ... and [1019 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...63ede5f](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r689805932
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -166,6 +171,16 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
When using a CountEvent that is the default name of the key used to store the count quantity if you see [here](https://jarvis.corp.linkedin.com/codesearch/result/?name=CountEventBuilder.java&path=gobblin-elr%2Fgobblin-metrics-libs%2Fgobblin-metrics-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fmetrics%2Fevent&reponame=linkedin%2Fgobblin-elr#59).
Edit: It doesn't look like I can rename it even with the reverse create the `CountEvent` from a `GobblinTrackingEvent` where I would be able to specify the key name by using the `fromEvent` [builder](https://jarvis.corp.linkedin.com/codesearch/result/?name=CountEventBuilder.java&path=gobblin-elr%2Fgobblin-metrics-libs%2Fgobblin-metrics-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fmetrics%2Fevent&reponame=linkedin%2Fgobblin-elr#74).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (63ede5f) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
=========================================
Coverage 46.43% 46.44%
+ Complexity 10120 10118 -2
=========================================
Files 2052 2052
Lines 79656 79666 +10
Branches 8896 8896
=========================================
+ Hits 36991 37001 +10
+ Misses 39226 39225 -1
- Partials 3439 3440 +1
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `27.77% <0.00%> (-5.56%)` | :arrow_down: |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | :arrow_down: |
| [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0.00%> (+1.56%)` | :arrow_up: |
| [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `84.67% <0.00%> (+2.41%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...63ede5f](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9fe511) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.42% 46.44% +0.01%
- Complexity 10115 10120 +5
============================================
Files 2052 2052
Lines 79626 79636 +10
Branches 8889 8889
============================================
+ Hits 36970 36986 +16
+ Misses 39216 39213 -3
+ Partials 3440 3437 -3
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [.../org/apache/gobblin/async/AsyncDataDispatcher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2FzeW5jL0FzeW5jRGF0YURpc3BhdGNoZXIuamF2YQ==) | `79.66% <0.00%> (-8.48%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0.00%> (+0.93%)` | :arrow_up: |
| [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0.00%> (+1.42%)` | :arrow_up: |
| [.../modules/scheduler/GobblinServiceJobScheduler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9zY2hlZHVsZXIvR29iYmxpblNlcnZpY2VKb2JTY2hlZHVsZXIuamF2YQ==) | `61.73% <0.00%> (+1.53%)` | :arrow_up: |
| [...gobblin/service/modules/core/GitConfigMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9jb3JlL0dpdENvbmZpZ01vbml0b3IuamF2YQ==) | `83.05% <0.00%> (+1.69%)` | :arrow_up: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `33.33% <0.00%> (+5.55%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...d9fe511](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] jack-moseley commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
jack-moseley commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r683766211
##########
File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
##########
@@ -205,6 +210,13 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOGGER.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_GROUP_KEY),
+ this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_NAME_KEY), "workunits");
Review comment:
Let's make the name a bit more descriptive than "workunits", maybe just reuse the `JobEvent.WORK_UNITS_CREATED` name that's used for the events.
##########
File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
##########
@@ -17,6 +17,8 @@
package org.apache.gobblin.cluster;
+import com.codahale.metrics.MetricRegistry;
Review comment:
Are you using https://gobblin.readthedocs.io/en/latest/developer-guide/CodingStyle/#code-style-template-file? Seems like the imports are not in the correct order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r687273985
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge guage = metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
The issue here is that we don't have access to the number of work units easily from the GaaS code. One option will be to use the metadata in the CountEvent like you are suggesting and then emit actual metrics from the point its landing in GaaS (the KafkaAvroJobStatusMonitor) but that seemed like a hackier way of doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] aplex commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r692535602
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -166,6 +171,15 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge gauge = this.getMetricContext().newContextAwareGauge(workUnitCountName, () -> numWorkUnits);
Review comment:
We can get "WORK_UNITS_CREATED" event a million times per day, and multiple times for same flow, if it runs repeatedly. I see that we are creating a newContextAwareGauge for each event. I wonder if this can result in a memory leak, when we keep adding gauges on each flow execution. Can you double-check how this newContextAwareGauge is supposed to be used (in docs, and in other places in code)? Should we get the existing one, if it's already present?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r689740222
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -166,6 +171,16 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
properties.getProperty(CountEventBuilder.COUNT_KEY) is telling how many workunits are created?
It could have been saved with a more self-explanatory name inside the workunit properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (63ede5f) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.40%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 43.02% -3.41%
+ Complexity 10120 1939 -8181
============================================
Files 2052 394 -1658
Lines 79656 16891 -62765
Branches 8896 2080 -6816
============================================
- Hits 36991 7268 -29723
+ Misses 39226 8821 -30405
+ Partials 3439 802 -2637
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...lin/runtime/template/ResourceBasedJobTemplate.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGVtcGxhdGUvUmVzb3VyY2VCYXNlZEpvYlRlbXBsYXRlLmphdmE=) | | |
| [...apache/gobblin/couchbase/common/TupleDocument.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS9jb21tb24vVHVwbGVEb2N1bWVudC5qYXZh) | | |
| [...che/gobblin/compaction/mapreduce/orc/OrcUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL29yYy9PcmNVdGlscy5qYXZh) | | |
| [...n/service/modules/orchestration/SessionHelper.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9zZXJ2aWNlL21vZHVsZXMvb3JjaGVzdHJhdGlvbi9TZXNzaW9uSGVscGVyLmphdmE=) | | |
| [...java/org/apache/gobblin/hive/HiveSerDeManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVTZXJEZU1hbmFnZXIuamF2YQ==) | | |
| [...nagement/copy/replication/ReplicationMetaData.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vUmVwbGljYXRpb25NZXRhRGF0YS5qYXZh) | | |
| [...nstrumented/writer/InstrumentedDataWriterBase.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5zdHJ1bWVudGVkL3dyaXRlci9JbnN0cnVtZW50ZWREYXRhV3JpdGVyQmFzZS5qYXZh) | | |
| ... and [1650 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...63ede5f](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] aplex commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r695254629
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ SortedMap<String, Gauge> existingGauges = this.getMetricContext().getGauges(
+ (name, metric) -> name.equals(workUnitCountName));
Review comment:
We are doing a full scan of metrics list here, so if there are 100k of them, it can have a performance impact. You can run a profiler on a large deployment to see if a lot of time is spent here. If it is, then we can add a method to check metric existence by name without scanning. Don't see such method in the current library.
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
There is a quite sizable chunk of logic in this "case" branch, you can move it to a separate method to make the current one smaller and easier to reason about.
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -64,6 +72,8 @@
@Getter
private Meter messageParseFailures;
+ private final HashMap<String, Long> flowNameGroupToWorkUnitCount;
Review comment:
I see that other fields in this class are ThreadLocal, which may indicate that this class can be called from multiple threads. Also, I'm pretty sure that the lambda " () -> this.flowNameGroupToWorkUnitCount.get(workUnitCountName)" below will be called by metrics library from a separate thread.
If a collection is accessed by multiple threads at the same time, you'll get occasional data corruption and strange exceptions.
You can use a thread-safe collection here instead from Apache commons library or Guava.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] aplex merged pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
aplex merged pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9fe511) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.42% 46.42% -0.01%
+ Complexity 10115 3072 -7043
============================================
Files 2052 644 -1408
Lines 79626 25096 -54530
Branches 8889 2996 -5893
============================================
- Hits 36970 11651 -25319
+ Misses 39216 12194 -27022
+ Partials 3440 1251 -2189
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...obblin/ingestion/google/webmaster/ProducerJob.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvUHJvZHVjZXJKb2IuamF2YQ==) | | |
| [...va/org/apache/gobblin/source/jdbc/JdbcCommand.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL0pkYmNDb21tYW5kLmphdmE=) | | |
| [...n/java/org/apache/gobblin/runtime/api/JobSpec.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYlNwZWMuamF2YQ==) | | |
| [...ava/org/apache/gobblin/runtime/ExecutionModel.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRXhlY3V0aW9uTW9kZWwuamF2YQ==) | | |
| [...org/apache/gobblin/metrics/test/MetricsAssert.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy90ZXN0L01ldHJpY3NBc3NlcnQuamF2YQ==) | | |
| [...va/org/apache/gobblin/converter/EmptyIterable.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9FbXB0eUl0ZXJhYmxlLmphdmE=) | | |
| [...ime/spec\_executorInstance/LocalFsSpecProducer.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19leGVjdXRvckluc3RhbmNlL0xvY2FsRnNTcGVjUHJvZHVjZXIuamF2YQ==) | | |
| ... and [1403 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...d9fe511](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4e70d52) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `1.73%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 48.17% +1.73%
+ Complexity 10120 7542 -2578
============================================
Files 2052 1423 -629
Lines 79656 56157 -23499
Branches 8896 6488 -2408
============================================
- Hits 36991 27055 -9936
+ Misses 39226 26564 -12662
+ Partials 3439 2538 -901
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `27.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `43.16% <0.00%> (-2.19%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...n/runtime/troubleshooter/JobIssueEventHandler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSm9iSXNzdWVFdmVudEhhbmRsZXIuamF2YQ==) | `70.27% <0.00%> (-1.96%)` | :arrow_down: |
| ... and [666 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...4e70d52](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4e70d52) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `2.51%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 48.95% +2.51%
+ Complexity 10120 8700 -1420
============================================
Files 2052 1676 -376
Lines 79656 64587 -15069
Branches 8896 7418 -1478
============================================
- Hits 36991 31617 -5374
+ Misses 39226 29976 -9250
+ Partials 3439 2994 -445
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `27.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `43.16% <0.00%> (-2.19%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| ... and [428 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...4e70d52](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r688788151
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
Review comment:
Moving to Gobblin Service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r683729121
##########
File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
##########
@@ -205,6 +209,12 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOGGER.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_GROUP_KEY),
+ this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_NAME_KEY), "workunits");
+ metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
Yes thank you, registered the metric. Here is the [PR](https://github.com/apache/gobblin/commit/6bed94a0f213ef64b225bb05b80ab338c25bbc17) relating to the register requirement for future reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r695043228
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -166,6 +171,15 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge gauge = this.getMetricContext().newContextAwareGauge(workUnitCountName, () -> numWorkUnits);
Review comment:
Changed this to obtain the existing gauge for a particular flow group + flow name if one exists and otherwise create a new gauge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (363c8e7) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `1.90%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 48.34% +1.90%
+ Complexity 10120 7517 -2603
============================================
Files 2052 1421 -631
Lines 79656 55889 -23767
Branches 8896 6439 -2457
============================================
- Hits 36991 27019 -9972
+ Misses 39226 26341 -12885
+ Partials 3439 2529 -910
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | :arrow_down: |
| [...odules/flowgraph/datanodes/fs/LocalFSDataNode.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvZGF0YW5vZGVzL2ZzL0xvY2FsRlNEYXRhTm9kZS5qYXZh) | | |
| [.../metrics/kafka/KafkaAvroEventKeyValueReporter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL21ldHJpY3Mva2Fma2EvS2Fma2FBdnJvRXZlbnRLZXlWYWx1ZVJlcG9ydGVyLmphdmE=) | | |
| [...gobblin/compliance/HivePartitionDatasetPolicy.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL0hpdmVQYXJ0aXRpb25EYXRhc2V0UG9saWN5LmphdmE=) | | |
| [...ava/org/apache/gobblin/http/ApacheHttpRequest.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9odHRwL0FwYWNoZUh0dHBSZXF1ZXN0LmphdmE=) | | |
| [...e/modules/orchestration/AzkabanSessionManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9zZXJ2aWNlL21vZHVsZXMvb3JjaGVzdHJhdGlvbi9BemthYmFuU2Vzc2lvbk1hbmFnZXIuamF2YQ==) | | |
| [...blin/compliance/purger/HivePurgerCommitPolicy.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL3B1cmdlci9IaXZlUHVyZ2VyQ29tbWl0UG9saWN5LmphdmE=) | | |
| [...bblin/service/modules/flowgraph/BaseFlowGraph.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvQmFzZUZsb3dHcmFwaC5qYXZh) | | |
| [...csearch/writer/ElasticsearchDataWriterBuilder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9FbGFzdGljc2VhcmNoRGF0YVdyaXRlckJ1aWxkZXIuamF2YQ==) | | |
| ... and [625 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...363c8e7](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r696146966
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ SortedMap<String, Gauge> existingGauges = this.getMetricContext().getGauges(
+ (name, metric) -> name.equals(workUnitCountName));
Review comment:
Created a [ticket](https://jira01.corp.linkedin.com:8443/browse/ETL-12789) to run a profiler on a large deployment and follow up with SRE's about impact of this metrics emission. I will add a method if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893038684
@arjun4084346 @jack-moseley can you review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9fe511) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **decrease** coverage by `3.38%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.42% 43.04% -3.39%
+ Complexity 10115 1942 -8173
============================================
Files 2052 394 -1658
Lines 79626 16882 -62744
Branches 8889 2076 -6813
============================================
- Hits 36970 7267 -29703
+ Misses 39216 8815 -30401
+ Partials 3440 800 -2640
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...gobblin/compaction/suite/CompactionSuiteUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vc3VpdGUvQ29tcGFjdGlvblN1aXRlVXRpbHMuamF2YQ==) | | |
| [...in/java/org/apache/gobblin/writer/RetryWriter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9SZXRyeVdyaXRlci5qYXZh) | | |
| [...ctor/extract/kafka/KafkaDeserializerExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL2V4dHJhY3Rvci9leHRyYWN0L2thZmthL0thZmthRGVzZXJpYWxpemVyRXh0cmFjdG9yLmphdmE=) | | |
| [...in/azkaban/AzkabanGobblinLocalYarnAppLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluTG9jYWxZYXJuQXBwTGF1bmNoZXIuamF2YQ==) | | |
| [...org/apache/gobblin/runtime/EventMetadataUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRXZlbnRNZXRhZGF0YVV0aWxzLmphdmE=) | | |
| [...gement/retention/version/StringDatasetVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3JldGVudGlvbi92ZXJzaW9uL1N0cmluZ0RhdGFzZXRWZXJzaW9uLmphdmE=) | | |
| [...apache/gobblin/runtime/api/SpecSerDeException.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL1NwZWNTZXJEZUV4Y2VwdGlvbi5qYXZh) | | |
| ... and [1651 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...d9fe511](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r689805932
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -166,6 +171,16 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
When using a CountEvent that is the default name of the key used to store the count quantity if you see [here](https://jarvis.corp.linkedin.com/codesearch/result/?name=CountEventBuilder.java&path=gobblin-elr%2Fgobblin-metrics-libs%2Fgobblin-metrics-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fmetrics%2Fevent&reponame=linkedin%2Fgobblin-elr#59). However, I could reverse create the `CountEvent` from a `GobblinTrackingEvent` where I would be able to specify the key name by using the `fromEvent` [builder](https://jarvis.corp.linkedin.com/codesearch/result/?name=CountEventBuilder.java&path=gobblin-elr%2Fgobblin-metrics-libs%2Fgobblin-metrics-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fmetrics%2Fevent&reponame=linkedin%2Fgobblin-elr#74). It would be a sort of roundabout way though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d9fe511) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **increase** coverage by `2.54%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.42% 48.97% +2.54%
+ Complexity 10115 8643 -1472
============================================
Files 2052 1671 -381
Lines 79626 64080 -15546
Branches 8889 7352 -1537
============================================
- Hits 36970 31386 -5584
+ Misses 39216 29714 -9502
+ Partials 3440 2980 -460
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [.../org/apache/gobblin/async/AsyncDataDispatcher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2FzeW5jL0FzeW5jRGF0YURpc3BhdGNoZXIuamF2YQ==) | `79.66% <0.00%> (-8.48%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...ce/extractor/extract/google/GoogleDriveSource.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL2V4dHJhY3Rvci9leHRyYWN0L2dvb2dsZS9Hb29nbGVEcml2ZVNvdXJjZS5qYXZh) | | |
| [...blin/compliance/purger/HivePurgerCommitPolicy.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY29tcGxpYW5jZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb21wbGlhbmNlL3B1cmdlci9IaXZlUHVyZ2VyQ29tbWl0UG9saWN5LmphdmE=) | | |
| [...rquet/JsonIntermediateToParquetGroupConverter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb252ZXJ0ZXIvcGFycXVldC9Kc29uSW50ZXJtZWRpYXRlVG9QYXJxdWV0R3JvdXBDb252ZXJ0ZXIuamF2YQ==) | | |
| [...lin/metadata/types/StaticStringMetadataMerger.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0YWRhdGEvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YWRhdGEvdHlwZXMvU3RhdGljU3RyaW5nTWV0YWRhdGFNZXJnZXIuamF2YQ==) | | |
| [...hbase/writer/CouchbaseWriterConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY291Y2hiYXNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvdWNoYmFzZS93cml0ZXIvQ291Y2hiYXNlV3JpdGVyQ29uZmlndXJhdGlvbktleXMuamF2YQ==) | | |
| ... and [381 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...d9fe511](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r683120849
##########
File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
##########
@@ -205,6 +209,12 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOGGER.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_GROUP_KEY),
+ this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_NAME_KEY), "workunits");
+ metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
You need to register this gauge to the metricContext, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a14fdb2) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.40%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 43.03% -3.41%
+ Complexity 10120 1957 -8163
============================================
Files 2052 395 -1657
Lines 79656 16952 -62704
Branches 8896 2090 -6806
============================================
- Hits 36991 7295 -29696
+ Misses 39226 8847 -30379
+ Partials 3439 810 -2629
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `32.23% <0.00%> (-2.48%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...org/apache/gobblin/iceberg/GobblinMCEProducer.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1pY2ViZXJnL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ljZWJlcmcvR29iYmxpbk1DRVByb2R1Y2VyLmphdmE=) | `48.71% <0.00%> (-1.29%)` | :arrow_down: |
| [...che/gobblin/metastore/DatabaseJobHistoryStore.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0RhdGFiYXNlSm9iSGlzdG9yeVN0b3JlLmphdmE=) | `77.77% <0.00%> (-0.49%)` | :arrow_down: |
| ... and [1665 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...a14fdb2](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (429639a) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 46.43% -0.01%
- Complexity 10120 10160 +40
============================================
Files 2052 2056 +4
Lines 79656 80094 +438
Branches 8896 8948 +52
============================================
+ Hits 36991 37190 +199
- Misses 39226 39458 +232
- Partials 3439 3446 +7
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `43.16% <0.00%> (-2.19%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...n/runtime/troubleshooter/JobIssueEventHandler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSm9iSXNzdWVFdmVudEhhbmRsZXIuamF2YQ==) | `70.27% <0.00%> (-1.96%)` | :arrow_down: |
| ... and [48 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...429639a](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-896320196
+1 LGTM but not sure about the necessity of this metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (429639a) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.34%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 43.09% -3.35%
+ Complexity 10120 1961 -8159
============================================
Files 2052 395 -1657
Lines 79656 16951 -62705
Branches 8896 2089 -6807
============================================
- Hits 36991 7305 -29686
+ Misses 39226 8837 -30389
+ Partials 3439 809 -2630
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...org/apache/gobblin/iceberg/GobblinMCEProducer.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1pY2ViZXJnL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ljZWJlcmcvR29iYmxpbk1DRVByb2R1Y2VyLmphdmE=) | `48.71% <0.00%> (-1.29%)` | :arrow_down: |
| [...che/gobblin/metastore/DatabaseJobHistoryStore.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0RhdGFiYXNlSm9iSGlzdG9yeVN0b3JlLmphdmE=) | `77.77% <0.00%> (-0.49%)` | :arrow_down: |
| [.../gobblin/cluster/HelixRetriggeringJobCallable.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhSZXRyaWdnZXJpbmdKb2JDYWxsYWJsZS5qYXZh) | `60.13% <0.00%> (-0.40%)` | :arrow_down: |
| [.../org/apache/gobblin/metastore/MysqlStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL015c3FsU3RhdGVTdG9yZS5qYXZh) | `8.16% <0.00%> (ø)` | |
| [...obblin/cluster/GobblinHelixJobLauncherMetrics.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXJNZXRyaWNzLmphdmE=) | `90.90% <0.00%> (ø)` | |
| [...che/gobblin/http/HttpClientConfiguratorLoader.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2h0dHAvSHR0cENsaWVudENvbmZpZ3VyYXRvckxvYWRlci5qYXZh) | | |
| ... and [1660 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...429639a](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0f9af55) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `7.62%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 54.06% +7.62%
+ Complexity 10120 1155 -8965
============================================
Files 2052 253 -1799
Lines 79656 8421 -71235
Branches 8896 929 -7967
============================================
- Hits 36991 4553 -32438
+ Misses 39226 3410 -35816
+ Partials 3439 458 -2981
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0.00% <ø> (ø)` | |
| [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/service/modules/orchestration/Orchestrator.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL09yY2hlc3RyYXRvci5qYXZh) | `52.17% <0.00%> (-0.94%)` | :arrow_down: |
| [...va/org/apache/gobblin/util/ClassAliasResolver.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdXRpbC9DbGFzc0FsaWFzUmVzb2x2ZXIuamF2YQ==) | `80.00% <0.00%> (ø)` | |
| [.../org/apache/gobblin/service/ServiceConfigKeys.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9TZXJ2aWNlQ29uZmlnS2V5cy5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../gobblin/runtime/CheckpointableWatermarkState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvZ29iYmxpbi9ydW50aW1lL0NoZWNrcG9pbnRhYmxlV2F0ZXJtYXJrU3RhdGUuamF2YQ==) | | |
| [...ata/management/copy/CopyableGlobDatasetFinder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQ29weWFibGVHbG9iRGF0YXNldEZpbmRlci5qYXZh) | | |
| [...g/apache/gobblin/runtime/NonRefillableLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTm9uUmVmaWxsYWJsZUxpbWl0ZXIuamF2YQ==) | | |
| [...che/gobblin/example/wikipedia/WikipediaSource.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1leGFtcGxlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2V4YW1wbGUvd2lraXBlZGlhL1dpa2lwZWRpYVNvdXJjZS5qYXZh) | | |
| [...lin/config/store/hdfs/SimpleHDFSStoreMetadata.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbmZpZy9zdG9yZS9oZGZzL1NpbXBsZUhERlNTdG9yZU1ldGFkYXRhLmphdmE=) | | |
| ... and [1801 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...0f9af55](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r696151126
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
Moved to a helper function in KafkaJobStatusMonitor and called it from KafkaJobStatusMonitor.processMessage() as you recommended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (16e5839) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **decrease** coverage by `3.42%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.42% 43.00% -3.43%
+ Complexity 10115 1938 -8177
============================================
Files 2052 394 -1658
Lines 79626 16882 -62744
Branches 8889 2076 -6813
============================================
- Hits 36970 7260 -29710
+ Misses 39216 8822 -30394
+ Partials 3440 800 -2640
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...li/GobblinServiceFlowExecutionResourceHandler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9yZXN0bGkvR29iYmxpblNlcnZpY2VGbG93RXhlY3V0aW9uUmVzb3VyY2VIYW5kbGVyLmphdmE=) | | |
| [...ta/management/dataset/SimpleFileSystemDataset.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2RhdGFzZXQvU2ltcGxlRmlsZVN5c3RlbURhdGFzZXQuamF2YQ==) | | |
| [...java/org/apache/gobblin/writer/ThrottleWriter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9UaHJvdHRsZVdyaXRlci5qYXZh) | | |
| [...kunit/packer/KafkaTopicGroupingWorkUnitPacker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS93b3JrdW5pdC9wYWNrZXIvS2Fma2FUb3BpY0dyb3VwaW5nV29ya1VuaXRQYWNrZXIuamF2YQ==) | | |
| [...obblin/source/extractor/hadoop/AvroFileSource.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvaGFkb29wL0F2cm9GaWxlU291cmNlLmphdmE=) | | |
| [...urce/extractor/extract/kafka/ConfigStoreUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9Db25maWdTdG9yZVV0aWxzLmphdmE=) | | |
| ... and [1651 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...16e5839](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r683120849
##########
File path: gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
##########
@@ -205,6 +209,12 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOGGER.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_GROUP_KEY),
+ this.jobContext.getJobState().getProp(ConfigurationKeys.FLOW_NAME_KEY), "workunits");
+ metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
You need to register this gauge to the metricContext, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (363c8e7) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.39%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 43.04% -3.40%
+ Complexity 10120 1942 -8178
============================================
Files 2052 394 -1658
Lines 79656 16886 -62770
Branches 8896 2080 -6816
============================================
- Hits 36991 7269 -29722
+ Misses 39226 8816 -30410
+ Partials 3439 801 -2638
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...e/gobblin/yarn/AbstractYarnAppSecurityManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vQWJzdHJhY3RZYXJuQXBwU2VjdXJpdHlNYW5hZ2VyLmphdmE=) | | |
| [.../apache/gobblin/source/jdbc/JdbcCommandOutput.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL0pkYmNDb21tYW5kT3V0cHV0LmphdmE=) | | |
| [...blin/restli/throttling/ThrottlingServerScopes.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1Rocm90dGxpbmdTZXJ2ZXJTY29wZXMuamF2YQ==) | | |
| [...compaction/verify/CompactionThresholdVerifier.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vdmVyaWZ5L0NvbXBhY3Rpb25UaHJlc2hvbGRWZXJpZmllci5qYXZh) | | |
| [.../copy/extractor/FileAwareInputStreamExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvZXh0cmFjdG9yL0ZpbGVBd2FyZUlucHV0U3RyZWFtRXh0cmFjdG9yLmphdmE=) | | |
| [...a/org/apache/gobblin/source/workunit/WorkUnit.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc291cmNlL3dvcmt1bml0L1dvcmtVbml0LmphdmE=) | | |
| [...blin/source/extractor/watermark/WatermarkType.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3Ivd2F0ZXJtYXJrL1dhdGVybWFya1R5cGUuamF2YQ==) | | |
| [...n/google/webmaster/GoogleWebMasterSourceDaily.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvR29vZ2xlV2ViTWFzdGVyU291cmNlRGFpbHkuamF2YQ==) | | |
| [...java/org/apache/gobblin/hive/HiveRegisterStep.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVSZWdpc3RlclN0ZXAuamF2YQ==) | | |
| [.../java/org/apache/gobblin/hive/HiveConfFactory.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL0hpdmVDb25mRmFjdG9yeS5qYXZh) | | |
| ... and [1648 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...363c8e7](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-896331225
@sv2000 can you merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-904903018
@aplex can you merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (63ede5f) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `0.03%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 46.40% -0.04%
+ Complexity 10120 3066 -7054
============================================
Files 2052 644 -1408
Lines 79656 25105 -54551
Branches 8896 3000 -5896
============================================
- Hits 36991 11649 -25342
+ Misses 39226 12201 -27025
+ Partials 3439 1255 -2184
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.81% <100.00%> (+0.44%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0.00%> (-7.15%)` | :arrow_down: |
| [...apache/gobblin/writer/AbstractAsyncDataWriter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3dyaXRlci9BYnN0cmFjdEFzeW5jRGF0YVdyaXRlci5qYXZh) | | |
| [...lin/runtime/job\_catalog/ImmutableFSJobCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvam9iX2NhdGFsb2cvSW1tdXRhYmxlRlNKb2JDYXRhbG9nLmphdmE=) | | |
| [...apache/gobblin/metrics/event/MultiTimingEvent.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9NdWx0aVRpbWluZ0V2ZW50LmphdmE=) | | |
| [...ce/extractor/DatePartitionedAvroFileExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvRGF0ZVBhcnRpdGlvbmVkQXZyb0ZpbGVFeHRyYWN0b3IuamF2YQ==) | | |
| [...main/java/org/apache/gobblin/types/TypeMapper.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdHlwZXMvVHlwZU1hcHBlci5qYXZh) | | |
| [...apache/gobblin/source/extractor/schema/Schema.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3Ivc2NoZW1hL1NjaGVtYS5qYXZh) | | |
| [...pache/gobblin/source/extractor/DummyExtractor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvRHVtbXlFeHRyYWN0b3IuamF2YQ==) | | |
| ... and [1400 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...63ede5f](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (429639a) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `1.75%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 48.19% +1.75%
+ Complexity 10120 7544 -2576
============================================
Files 2052 1423 -629
Lines 79656 56153 -23503
Branches 8896 6487 -2409
============================================
- Hits 36991 27062 -9929
+ Misses 39226 26555 -12671
+ Partials 3439 2536 -903
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `43.16% <0.00%> (-2.19%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...n/runtime/troubleshooter/JobIssueEventHandler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSm9iSXNzdWVFdmVudEhhbmRsZXIuamF2YQ==) | `70.27% <0.00%> (-1.96%)` | :arrow_down: |
| [.../gobblin/runtime/spec\_catalog/TopologyCatalog.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvc3BlY19jYXRhbG9nL1RvcG9sb2d5Q2F0YWxvZy5qYXZh) | `54.08% <0.00%> (-1.71%)` | :arrow_down: |
| ... and [663 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...429639a](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4e70d52) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `0.01%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 46.42% -0.02%
- Complexity 10120 10159 +39
============================================
Files 2052 2056 +4
Lines 79656 80097 +441
Branches 8896 8949 +53
============================================
+ Hits 36991 37184 +193
- Misses 39226 39466 +240
- Partials 3439 3447 +8
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `61.11% <ø> (-0.15%)` | :arrow_down: |
| [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...leshooter/InMemoryMultiContextIssueRepository.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdHJvdWJsZXNob290ZXIvSW5NZW1vcnlNdWx0aUNvbnRleHRJc3N1ZVJlcG9zaXRvcnkuamF2YQ==) | `60.00% <0.00%> (-35.24%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...e/gobblin/runtime/local/LocalTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9jYWwvTG9jYWxUYXNrU3RhdGVUcmFja2VyLmphdmE=) | `72.97% <0.00%> (-7.59%)` | :arrow_down: |
| [...t/version/TimestampedDatasetStateStoreVersion.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L3ZlcnNpb24vVGltZXN0YW1wZWREYXRhc2V0U3RhdGVTdG9yZVZlcnNpb24uamF2YQ==) | `27.77% <0.00%> (-5.56%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `43.16% <0.00%> (-2.19%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| ... and [50 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...4e70d52](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (363c8e7) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `2.54%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
+ Coverage 46.43% 48.98% +2.54%
+ Complexity 10120 8646 -1474
============================================
Files 2052 1671 -381
Lines 79656 64103 -15553
Branches 8896 7359 -1537
============================================
- Hits 36991 31401 -5590
+ Misses 39226 29722 -9504
+ Partials 3439 2980 -459
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70.00% <0.00%> (-2.23%)` | :arrow_down: |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | :arrow_down: |
| [...rquet/JsonIntermediateToParquetGroupConverter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC1hcGFjaGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29udmVydGVyL3BhcnF1ZXQvSnNvbkludGVybWVkaWF0ZVRvUGFycXVldEdyb3VwQ29udmVydGVyLmphdmE=) | | |
| [.../converter/avro/AvroToJsonStringConverterBase.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXZyby1qc29uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0F2cm9Ub0pzb25TdHJpbmdDb252ZXJ0ZXJCYXNlLmphdmE=) | | |
| [...estion/google/webmaster/GoogleWebMasterSource.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvR29vZ2xlV2ViTWFzdGVyU291cmNlLmphdmE=) | | |
| [.../gobblin/kafka/serialize/GsonDeserializerBase.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3NlcmlhbGl6ZS9Hc29uRGVzZXJpYWxpemVyQmFzZS5qYXZh) | | |
| [...va/org/apache/gobblin/type/RecordWithMetadata.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0YWRhdGEvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vdHlwZS9SZWNvcmRXaXRoTWV0YWRhdGEuamF2YQ==) | | |
| [...tadata/provider/SimpleMetadataProviderFactory.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tbWV0YWRhdGEvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YWRhdGEvcHJvdmlkZXIvU2ltcGxlTWV0YWRhdGFQcm92aWRlckZhY3RvcnkuamF2YQ==) | | |
| [...in/kafka/schemareg/CachingKafkaSchemaRegistry.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3NjaGVtYXJlZy9DYWNoaW5nS2Fma2FTY2hlbWFSZWdpc3RyeS5qYXZh) | | |
| ... and [377 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...363c8e7](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (363c8e7) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
=========================================
Coverage 46.43% 46.44%
- Complexity 10120 10123 +3
=========================================
Files 2052 2052
Lines 79656 79661 +5
Branches 8896 8896
=========================================
+ Hits 36991 37000 +9
+ Misses 39226 39223 -3
+ Partials 3439 3438 -1
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.12% <100.00%> (+0.56%)` | :arrow_up: |
| [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70.00% <0.00%> (-2.23%)` | :arrow_down: |
| [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `61.42% <0.00%> (-1.43%)` | :arrow_down: |
| [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | :arrow_down: |
| [.../modules/scheduler/GobblinServiceJobScheduler.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9zY2hlZHVsZXIvR29iYmxpblNlcnZpY2VKb2JTY2hlZHVsZXIuamF2YQ==) | `61.73% <0.00%> (+1.53%)` | :arrow_up: |
| [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0.00%> (+1.56%)` | :arrow_up: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.01% <0.00%> (+3.30%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...363c8e7](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4e70d52) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.39%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 43.03% -3.40%
+ Complexity 10120 1960 -8160
============================================
Files 2052 395 -1657
Lines 79656 16952 -62704
Branches 8896 2090 -6806
============================================
- Hits 36991 7296 -29695
+ Misses 39226 8846 -30380
+ Partials 3439 810 -2629
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...org/apache/gobblin/iceberg/GobblinMCEProducer.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1pY2ViZXJnL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ljZWJlcmcvR29iYmxpbk1DRVByb2R1Y2VyLmphdmE=) | `48.71% <0.00%> (-1.29%)` | :arrow_down: |
| [.../apache/gobblin/cluster/GobblinClusterManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkNsdXN0ZXJNYW5hZ2VyLmphdmE=) | `52.99% <0.00%> (-0.50%)` | :arrow_down: |
| [...che/gobblin/metastore/DatabaseJobHistoryStore.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL0RhdGFiYXNlSm9iSGlzdG9yeVN0b3JlLmphdmE=) | `77.77% <0.00%> (-0.49%)` | :arrow_down: |
| [.../gobblin/cluster/HelixRetriggeringJobCallable.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhSZXRyaWdnZXJpbmdKb2JDYWxsYWJsZS5qYXZh) | `60.13% <0.00%> (-0.40%)` | :arrow_down: |
| [.../org/apache/gobblin/metastore/MysqlStateStore.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL015c3FsU3RhdGVTdG9yZS5qYXZh) | `8.16% <0.00%> (ø)` | |
| ... and [1660 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...4e70d52](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f8aaa7b) into [master](https://codecov.io/gh/apache/gobblin/commit/f468bb0743a2d2f3de65abc2bd1a8cc7994c16fd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f468bb0) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
=========================================
Coverage 46.42% 46.43%
+ Complexity 10115 10113 -2
=========================================
Files 2052 2052
Lines 79626 79634 +8
Branches 8889 8889
=========================================
+ Hits 36970 36976 +6
+ Misses 39216 39215 -1
- Partials 3440 3443 +3
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.73% <100.00%> (+0.35%)` | :arrow_up: |
| [...pache/gobblin/runtime/mapreduce/MRJobLauncher.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL01SSm9iTGF1bmNoZXIuamF2YQ==) | `54.01% <100.00%> (+0.45%)` | :arrow_up: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `60.21% <0.00%> (-2.16%)` | :arrow_down: |
| [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `63.46% <0.00%> (-0.33%)` | :arrow_down: |
| [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0.00%> (+1.42%)` | :arrow_up: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.71% <0.00%> (+2.47%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f468bb0...f8aaa7b](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] aplex commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r695942214
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +177,26 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
Review comment:
Just noticed that current method is called "parseJobStatus", and looks like we are doing something extra here. You can also consider moving this logic to KafkaJobStatusMonitor.processMessage() , which can be more fitting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] aplex commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
aplex commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r687241795
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
Review comment:
FLOW_GROUP / FLOW_NAME can be null when the Gobblin job is executed outside of Gobblin service (just as Azkaban flow). This also means that "ServiceMetricNames.GOBBLIN_SERVICE_PREFIX" would not be applicable to such jobs.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
Review comment:
The base class has optional runtimeMetricContext. Can we use that? I think it will add some extra tags to metrics to tell them apart across the job.
##########
File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java
##########
@@ -291,6 +296,14 @@ protected void runWorkUnits(List<WorkUnit> workUnits) throws Exception {
this.eventSubmitter.submit(countEventBuilder);
LOG.info("Emitting WorkUnitsCreated Count: " + countEventBuilder.getCount());
+ MetricContext
+ metricContext = Instrumented.getMetricContext(ConfigUtils.configToState(ConfigFactory.empty()), getClass());
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ jobState.getProp(ConfigurationKeys.FLOW_GROUP_KEY), jobState.getProp(ConfigurationKeys.FLOW_NAME_KEY),
+ JobEvent.WORK_UNITS_CREATED);
+ ContextAwareGauge guage = metricContext.newContextAwareGauge(workUnitCountName, () -> workUnits.size());
Review comment:
Where do we emit other metrics for GaaS-triggered jobs - inside the job itself or in the GaaS code?
I see that we are emitting CountEvent for work unit size earlier in the code. I assume it will land in GaaS and be available for processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r694420139
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +179,30 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ MetricFilter filter = new MetricFilter() {
Review comment:
this can be outside the switch case block
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +179,30 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ MetricFilter filter = new MetricFilter() {
Review comment:
this should be outside the switch case block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] arjun4084346 commented on a change in pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
arjun4084346 commented on a change in pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#discussion_r694422827
##########
File path: gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/KafkaAvroJobStatusMonitor.java
##########
@@ -165,6 +179,30 @@ public GobblinTrackingEvent deserializeEvent(DecodeableKafkaRecord<byte[],byte[]
break;
case TimingEvent.JOB_COMPLETION_PERCENTAGE:
properties.put(TimingEvent.JOB_LAST_PROGRESS_EVENT_TIME, properties.getProperty(TimingEvent.METADATA_END_TIME));
+ break;
+ case JobEvent.WORK_UNITS_CREATED:
+ Long numWorkUnits = Long.parseLong(properties.getProperty(CountEventBuilder.COUNT_KEY));
+ String workUnitCountName = MetricRegistry.name(ServiceMetricNames.GOBBLIN_SERVICE_PREFIX,
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_GROUP_FIELD),
+ properties.getProperty(TimingEvent.FlowEventConstants.FLOW_NAME_FIELD),
+ JobEvent.WORK_UNITS_CREATED);
+
+ MetricFilter filter = new MetricFilter() {
+ @Override public boolean matches(String name, Metric metric) {
+ return name.equals(workUnitCountName);
+ }
+ };
+ SortedMap<String, Gauge> existingGauges = this.getMetricContext().getGauges(filter);
Review comment:
*nit
SortedMap<String, Gauge> existingGauges = this.getMetricContext().getGauges((name, metric) -> name.equals(workUnitCountName));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] umustafi commented on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
umustafi commented on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893038684
@arjun4084346 @jack-moseley can you review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter edited a comment on pull request #3351: [GOBBLIN-1505] Add metrics for WorkUnit count
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #3351:
URL: https://github.com/apache/gobblin/pull/3351#issuecomment-893035078
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3351](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a14fdb2) into [master](https://codecov.io/gh/apache/gobblin/commit/de94515d04ea615bdff86b306f34f5361e1725e6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (de94515) will **decrease** coverage by `3.83%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #3351 +/- ##
============================================
- Coverage 46.43% 42.60% -3.84%
+ Complexity 10120 4573 -5547
============================================
Files 2052 1028 -1024
Lines 79656 40897 -38759
Branches 8896 4552 -4344
============================================
- Hits 36991 17424 -19567
+ Misses 39226 21754 -17472
+ Partials 3439 1719 -1720
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...blin/service/monitoring/KafkaJobStatusMonitor.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUpvYlN0YXR1c01vbml0b3IuamF2YQ==) | `35.57% <0.00%> (-7.45%)` | :arrow_down: |
| [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40.00% <0.00%> (-20.00%)` | :arrow_down: |
| [...g/apache/gobblin/util/filters/RegexPathFilter.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsdGVycy9SZWdleFBhdGhGaWx0ZXIuamF2YQ==) | `72.72% <0.00%> (-16.17%)` | :arrow_down: |
| [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0.00%> (-14.29%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `32.23% <0.00%> (-2.48%)` | :arrow_down: |
| [...ore/util/DatabaseJobHistoryStoreSchemaManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tZXRhc3RvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0YXN0b3JlL3V0aWwvRGF0YWJhc2VKb2JIaXN0b3J5U3RvcmVTY2hlbWFNYW5hZ2VyLmphdmE=) | `29.31% <0.00%> (-2.36%)` | :arrow_down: |
| [...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh) | `76.62% <0.00%> (-2.05%)` | :arrow_down: |
| [.../gobblin/cluster/GobblinHelixTaskStateTracker.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFza1N0YXRlVHJhY2tlci5qYXZh) | `62.50% <0.00%> (-2.02%)` | :arrow_down: |
| [...ache/gobblin/cluster/GobblinHelixMultiManager.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4TXVsdGlNYW5hZ2VyLmphdmE=) | `52.33% <0.00%> (-1.84%)` | :arrow_down: |
| [...org/apache/gobblin/iceberg/GobblinMCEProducer.java](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1pY2ViZXJnL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ljZWJlcmcvR29iYmxpbk1DRVByb2R1Y2VyLmphdmE=) | `48.71% <0.00%> (-1.29%)` | :arrow_down: |
| ... and [1049 more](https://codecov.io/gh/apache/gobblin/pull/3351/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [de94515...a14fdb2](https://codecov.io/gh/apache/gobblin/pull/3351?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org