You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by GitBox <gi...@apache.org> on 2022/05/23 12:10:46 UTC
[GitHub] [incubator-linkis] Alexkun opened a new pull request, #2163: record EC information
Alexkun opened a new pull request, #2163:
URL: https://github.com/apache/incubator-linkis/pull/2163
### What is the purpose of the change
issue #2075
### Brief change log
(for example:)
- Optimize TaskYarnResourceEvent and TaskProgressEvent as one Event, defined as TaskRunningInfoEvent
- Record ec task progress info、ec resource info、ec extension info
### Verifying this change
This change added tests and can be verified as follows:
(example:)
- Submit any task to see the metrics in table linkis_ps_job_history_group_history
### Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- Anything that affects deployment: (no)
- The MGS(Microservice Governance Services), i.e., Spring Cloud Gateway, OpenFeign, Eureka.: (no)
### Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] codecov[bot] commented on pull request #2163: record EC information
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #2163:
URL: https://github.com/apache/incubator-linkis/pull/2163#issuecomment-1134622770
# [Codecov](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2163](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (75ebd5b) into [dev-1.1.3](https://codecov.io/gh/apache/incubator-linkis/commit/f4614b5ad1d8c7b774e8db5c27ca79760ea91e56?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f4614b5) will **increase** coverage by `1.15%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## dev-1.1.3 #2163 +/- ##
===============================================
+ Coverage 18.45% 19.61% +1.15%
- Complexity 703 843 +140
===============================================
Files 344 384 +40
Lines 10723 11825 +1102
Branches 1608 1757 +149
===============================================
+ Hits 1979 2319 +340
- Misses 8539 9263 +724
- Partials 205 243 +38
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...governance/common/constant/job/TaskConstants.scala](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLWNvbXB1dGF0aW9uLWdvdmVybmFuY2UvbGlua2lzLWNvbXB1dGF0aW9uLWdvdmVybmFuY2UtY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGlua2lzL2dvdmVybmFuY2UvY29tbW9uL2NvbnN0YW50L2pvYi9UYXNrQ29uc3RhbnRzLnNjYWxh) | `0.00% <0.00%> (ø)` | |
| [...org/apache/linkis/jobhistory/util/QueryUtils.scala](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWpvYmhpc3Rvcnkvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9saW5raXMvam9iaGlzdG9yeS91dGlsL1F1ZXJ5VXRpbHMuc2NhbGE=) | `17.64% <0.00%> (-0.27%)` | :arrow_down: |
| [...linkis/configuration/service/CategoryService.scala](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWNvbmZpZ3VyYXRpb24vc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9saW5raXMvY29uZmlndXJhdGlvbi9zZXJ2aWNlL0NhdGVnb3J5U2VydmljZS5zY2FsYQ==) | `40.76% <0.00%> (ø)` | |
| [...che/linkis/configuration/entity/CategoryLabel.java](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWNvbmZpZ3VyYXRpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpbmtpcy9jb25maWd1cmF0aW9uL2VudGl0eS9DYXRlZ29yeUxhYmVsLmphdmE=) | `84.61% <0.00%> (ø)` | |
| [...nfiguration/service/impl/ConfigKeyServiceImpl.java](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWNvbmZpZ3VyYXRpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpbmtpcy9jb25maWd1cmF0aW9uL3NlcnZpY2UvaW1wbC9Db25maWdLZXlTZXJ2aWNlSW1wbC5qYXZh) | `3.57% <0.00%> (ø)` | |
| [...linkis/configuration/validate/FloatValidator.scala](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWNvbmZpZ3VyYXRpb24vc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9saW5raXMvY29uZmlndXJhdGlvbi92YWxpZGF0ZS9GbG9hdFZhbGlkYXRvci5zY2FsYQ==) | `11.11% <0.00%> (ø)` | |
| [.../linkis/variable/service/VariableServiceImpl.scala](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLXZhcmlhYmxlL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvbGlua2lzL3ZhcmlhYmxlL3NlcnZpY2UvVmFyaWFibGVTZXJ2aWNlSW1wbC5zY2FsYQ==) | `28.00% <0.00%> (ø)` | |
| [...e/linkis/variable/exception/VariableException.java](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLXZhcmlhYmxlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saW5raXMvdmFyaWFibGUvZXhjZXB0aW9uL1ZhcmlhYmxlRXhjZXB0aW9uLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...apache/linkis/configuration/entity/ConfigTree.java](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLWNvbmZpZ3VyYXRpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2xpbmtpcy9jb25maWd1cmF0aW9uL2VudGl0eS9Db25maWdUcmVlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../apache/linkis/variable/entity/VarKeyAndValue.java](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-bGlua2lzLXB1YmxpYy1lbmhhbmNlbWVudHMvbGlua2lzLXB1YmxpY3NlcnZpY2UvbGlua2lzLXZhcmlhYmxlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9saW5raXMvdmFyaWFibGUvZW50aXR5L1ZhcktleUFuZFZhbHVlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [31 more](https://codecov.io/gh/apache/incubator-linkis/pull/2163/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f4614b5...75ebd5b](https://codecov.io/gh/apache/incubator-linkis/pull/2163?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong merged pull request #2163: record EC information
Posted by GitBox <gi...@apache.org>.
peacewong merged PR #2163:
URL: https://github.com/apache/incubator-linkis/pull/2163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org
[GitHub] [incubator-linkis] peacewong commented on a diff in pull request #2163: record EC information
Posted by GitBox <gi...@apache.org>.
peacewong commented on code in PR #2163:
URL: https://github.com/apache/incubator-linkis/pull/2163#discussion_r891013800
##########
linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java:
##########
@@ -294,43 +294,48 @@ public Message progressWithResource(@PathVariable("id") String id) {
(HashMap<String, ResourceWithStatus>)
metrics.get(TaskConstant.ENTRANCEJOB_YARNRESOURCE);
ArrayList<YarnResourceWithStatusVo> resoureList = new ArrayList<>(12);
- resourceMap.forEach(
- (applicationId, resource) -> {
- resoureList.add(
- new YarnResourceWithStatusVo(applicationId, resource));
- });
- metricsVo.put(TaskConstant.ENTRANCEJOB_YARNRESOURCE, resoureList);
- Optional<Integer> cores =
- resourceMap.values().stream()
- .map(resource -> resource.queueCores())
- .reduce((x, y) -> x + y);
- Optional<Long> memory =
- resourceMap.values().stream()
- .map(resource -> resource.queueMemory())
- .reduce((x, y) -> x + y);
- float corePercent = 0.0f;
- float memoryPercent = 0.0f;
- if (cores.isPresent()) {
- corePercent =
- cores.get().floatValue()
- / EntranceConfiguration.YARN_QUEUE_CORES_MAX().getValue();
- memoryPercent =
- memory.get().floatValue()
- / (EntranceConfiguration.YARN_QUEUE_MEMORY_MAX()
- .getValue()
- .longValue()
- * 1024
- * 1024
- * 1024);
- }
- String coreRGB = RGBUtils.getRGB(corePercent);
- String memoryRGB = RGBUtils.getRGB(memoryPercent);
- metricsVo.put(TaskConstant.ENTRANCEJOB_CORE_PERCENT, corePercent);
- metricsVo.put(TaskConstant.ENTRANCEJOB_MEMORY_PERCENT, memoryPercent);
- metricsVo.put(TaskConstant.ENTRANCEJOB_CORE_RGB, coreRGB);
- metricsVo.put(TaskConstant.ENTRANCEJOB_MEMORY_RGB, memoryRGB);
+ if (null != resourceMap && !resourceMap.isEmpty()) {
+ resourceMap.forEach(
+ (applicationId, resource) -> {
+ resoureList.add(
+ new YarnResourceWithStatusVo(applicationId, resource));
+ });
+ metricsVo.put(TaskConstant.ENTRANCEJOB_YARNRESOURCE, resoureList);
+ Optional<Integer> cores =
+ resourceMap.values().stream()
+ .map(resource -> resource.queueCores())
+ .reduce((x, y) -> x + y);
+ Optional<Long> memory =
+ resourceMap.values().stream()
+ .map(resource -> resource.queueMemory())
+ .reduce((x, y) -> x + y);
+ float corePercent = 0.0f;
+ float memoryPercent = 0.0f;
+ if (cores.isPresent()) {
+ corePercent =
+ cores.get().floatValue()
+ / EntranceConfiguration.YARN_QUEUE_CORES_MAX()
+ .getValue();
+ memoryPercent =
Review Comment:
Need to determine whether the memory isPresent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org