You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "Radeity (via GitHub)" <gi...@apache.org> on 2023/04/09 15:45:36 UTC

[GitHub] [dolphinscheduler] Radeity commented on a diff in pull request #13896: Use percentage to represent memory/cpu usage

Radeity commented on code in PR #13896:
URL: https://github.com/apache/dolphinscheduler/pull/13896#discussion_r1161304585


##########
dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/task/WorkerHeartBeatTask.java:
##########
@@ -90,16 +88,16 @@ public void writeHeartBeat(WorkerHeartBeat workerHeartBeat) {
                 workerRegistryPath, workerHeartBeatJson);
     }
 
-    public int getServerStatus(double loadAverage,
-                               double maxCpuloadAvg,
+    public int getServerStatus(double cpuUsagePercentage,
+                               double maxCpuUsePercentage,
                                double availablePhysicalMemorySize,
                                double reservedMemory,
                                int workerExecThreadCount,
                                int workerWaitingTaskCount) {
-        if (loadAverage > maxCpuloadAvg || availablePhysicalMemorySize < reservedMemory) {
+        if (cpuUsagePercentage > maxCpuUsePercentage || availablePhysicalMemorySize < reservedMemory) {

Review Comment:
   Here, it still compares the physical memory with reserved memory, their unit are both GB.



##########
dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/utils/OSUtils.java:
##########
@@ -466,23 +433,25 @@ public static int getProcessID() {
     /**
      * Check memory and cpu usage is overload the given thredshod.
      *
-     * @param maxCpuLoadAvg  maxCpuLoadAvg
-     * @param reservedMemory reservedMemory
+     * @param maxCpuLoadAvgThreshold  maxCpuLoadAvg
+     * @param reservedMemoryThreshold reservedMemory
      * @return True, if the cpu or memory exceed the given thredshod.
      */
-    public static Boolean isOverload(double maxCpuLoadAvg, double reservedMemory) {
+    public static Boolean isOverload(double maxCpuLoadAvgThreshold, double reservedMemoryThreshold) {
         // system load average
-        double loadAverage = loadAverage();
+        double freeCPUPercentage = 1 - cpuUsagePercentage();
         // system available physical memory
-        double availablePhysicalMemorySize = availablePhysicalMemorySize();
-        if (loadAverage > maxCpuLoadAvg) {
-            log.warn("Current cpu load average {} is too high, max.cpuLoad.avg={}", loadAverage, maxCpuLoadAvg);
+        double freeMemoryPercentage = 1 - memoryUsagePercentage();
+        if (freeCPUPercentage > maxCpuLoadAvgThreshold) {
+            log.warn("Current cpu load average {} is too high, max.cpuLoad.avg={}", freeCPUPercentage,
+                    maxCpuLoadAvgThreshold);
             return true;
         }
 
-        if (availablePhysicalMemorySize < reservedMemory) {
+        if (freeMemoryPercentage < reservedMemoryThreshold) {
             log.warn(
-                    "Current available memory {}G is too low, reserved.memory={}G", maxCpuLoadAvg, reservedMemory);
+                    "Current available memory percentage{} is too low, reserved.memory={}", freeMemoryPercentage,
+                    reservedMemoryThreshold);

Review Comment:
   The unit of `reservedMemory ` is still GB, if you want to change the memory and cpu representation to percentage, you may have to also modify the check in `getServerStatus `.



##########
dolphinscheduler-worker/src/main/resources/application.yaml:
##########
@@ -48,9 +48,9 @@ worker:
   tenant-auto-create: true
   #Scenes to be used for distributed users.For example,users created by FreeIpa are stored in LDAP.This parameter only applies to Linux, When this parameter is true, worker.tenant.auto.create has no effect and will not automatically create tenants.
   tenant-distributed-user: false
-  # worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks. default value -1: the number of cpu cores * 2
-  max-cpu-load-avg: -1
-  # worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, the unit is G
+  # worker max cpuload avg, only higher than the system cpu load average, worker server can be dispatched tasks. default value 1: will use 100% cpu.
+  max-cpu-load-avg: 1
+  # worker reserved memory, only lower than system available memory, worker server can be dispatched tasks. default value 0.3, only the available memory is higher than 30%, worker server can receive task.
   reserved-memory: 0.3

Review Comment:
   Maybe you can modify the doc of `reserved-memory`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org