You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/04/02 04:10:27 UTC
[GitHub] [incubator-dolphinscheduler] chengshiwen opened a new issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
chengshiwen opened a new issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198
**Bug description**
There are several metric methods: `loadAverage`, `cpuUsage`, `memoryUsage`, `totalMemorySize`, and `availablePhysicalMemorySize` in the file [OSUtils.java](https://github.com/apache/incubator-dolphinscheduler/blob/dev/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/OSUtils.java), and
[Runtime.getRuntime().availableProcessors()](https://github.com/apache/incubator-dolphinscheduler/blob/e8c9c33d7e45b361c3c25085b01d3d78b6b11d90/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java#L381) is also used.
In physical or virtual machines, these metrics can be collected correctly. But in docker/k8s container, these metrics are collected from the physical or virtual host. **In other words, these metrics in docker/k8s container are incorrect**.
**Reproduce step**
1. `docker: --cpus 3 --memory 2G`
```
JAVA_OPTS: -Xms1g -Xmx1g -Xmn512m -XshowSettings:vm
VM settings:
Min. Heap Size: 1.00G
Max. Heap Size: 1.00G
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
Runtime.getRuntime().availableProcessors: 4
Runtime.getRuntime().totalMemory: 960M
Runtime.getRuntime().freeMemory: 944M
Runtime.getRuntime().maxMemory: 960M
Oshi.Memory.getTotal: 3947M
Oshi.Memory.getAvailable: 669M
Oshi.Memory.getSwapUsed: 55M
Oshi.Memory.getSwapTotal: 1535M
Oshi.Processor.getLogicalProcessorCount: 4
Oshi.Processor.getPhysicalProcessorCount: 4
Oshi.Processor.getSystemCpuLoad: 0.0
Oshi.Processor.getSystemLoadAverage: 0.41
Oshi.Processor.getSystemCpuLoadBetweenTicks: 0.018198487340747176
OSUtils.loadAverage: 0.41
OSUtils.cpuUsage: 0.0
OSUtils.memoryUsage: 0.83
OSUtils.totalMemorySize: 3.85G
OSUtils.availablePhysicalMemorySize: 0.65G
```
2. `kubernetes: --requests cpu=500m,memory=512Mi --limits cpu=3,memory=2Gi`
```
JAVA_OPTS: -Xms1g -Xmx1g -Xmn512m -XshowSettings:vm
VM settings:
Min. Heap Size: 1.00G
Max. Heap Size: 1.00G
Ergonomics Machine Class: server
Using VM: OpenJDK 64-Bit Server VM
Runtime.getRuntime().availableProcessors: 16
Runtime.getRuntime().totalMemory: 960M
Runtime.getRuntime().freeMemory: 944M
Runtime.getRuntime().maxMemory: 960M
Oshi.Memory.getTotal: 30985M
Oshi.Memory.getAvailable: 19891M
Oshi.Memory.getSwapUsed: 0M
Oshi.Memory.getSwapTotal: 0M
Oshi.Processor.getLogicalProcessorCount: 16
Oshi.Processor.getPhysicalProcessorCount: 8
Oshi.Processor.getSystemCpuLoad: 0.0
Oshi.Processor.getSystemLoadAverage: 0.89
Oshi.Processor.getSystemCpuLoadBetweenTicks: 0.0445536013301334
OSUtils.loadAverage: 0.89
OSUtils.cpuUsage: 0.0
OSUtils.memoryUsage: 0.36
OSUtils.totalMemorySize: 30.26G
OSUtils.availablePhysicalMemorySize: 19.43G
```
**Expected behavior**
Bug fixed.
**Which version of Dolphin Scheduler:**
-[1.3.x]
-[dev]
**Additional context**
Add any other context about the problem here.
We can refer to [Container runtime metrics](https://docs.docker.com/config/containers/runmetrics/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] cutiechi commented on issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
Posted by GitBox <gi...@apache.org>.
cutiechi commented on issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198#issuecomment-818871935
@chengshiwen What is your Java version? When I use the latest image (container support is enabled by default), these results are correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198#issuecomment-819235178
> @chengshiwen What is your Java version? When I use the latest image (container support is enabled by default), these results are correct.
```
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
```
And `these results are correct` you mentiond are **not all correct**.
`UseContainerSupport` is not friendly to memory in Docker/K8s and `Runtime.getRuntime().availableProcessors` in K8s. You will see `Runtime.getRuntime().availableProcessors=1` if `requests.limts.cpu` is not configuredin K8s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198#issuecomment-819235178
> @chengshiwen What is your Java version? When I use the latest image (container support is enabled by default), these results are correct.
```
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
```
`UseContainerSupport` is not friendly to memory in Docker/K8s and `Runtime.getRuntime().availableProcessors` in K8s. You will see `Runtime.getRuntime().availableProcessors=1` if `requests.limts.cpu` is not configuredin K8s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen edited a comment on issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
Posted by GitBox <gi...@apache.org>.
chengshiwen edited a comment on issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198#issuecomment-819235178
> @chengshiwen What is your Java version? When I use the latest image (container support is enabled by default), these results are correct.
```
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
```
And `these results are correct` you mentioned are **not all correct**.
`UseContainerSupport` is not friendly to memory in Docker/K8s and `Runtime.getRuntime().availableProcessors` in K8s. You will see `Runtime.getRuntime().availableProcessors=1` if `requests.limts.cpu` is not configuredin K8s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-dolphinscheduler] chengshiwen commented on issue #5198: [Bug][Docker/K8s] Incorrect runtime metrics of cpu and memory in Docker/K8s
Posted by GitBox <gi...@apache.org>.
chengshiwen commented on issue #5198:
URL: https://github.com/apache/incubator-dolphinscheduler/issues/5198#issuecomment-819235178
> @chengshiwen What is your Java version? When I use the latest image (container support is enabled by default), these results are correct.
>
> 
```
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
```
`UseContainerSupport` is not friendly to memory in Docker/K8s and `Runtime.getRuntime().availableProcessors` in K8s. You will see `Runtime.getRuntime().availableProcessors=1` if `requests.limts.cpu` is not configuredin K8s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org