You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/07/08 01:09:06 UTC

[GitHub] [doris] Mryange opened a new pull request, #21640: [refactor](planner) refactor automatically set instance_num

Mryange opened a new pull request, #21640:
URL: https://github.com/apache/doris/pull/21640

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei commented on a diff in pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #21640:
URL: https://github.com/apache/doris/pull/21640#discussion_r1256716909


##########
fe/fe-core/src/main/java/org/apache/doris/system/Backend.java:
##########
@@ -519,6 +542,24 @@ public void updateDisks(Map<String, TDisk> backendDisks) {
         }
     }
 
+    public void updateCpuInfo(int cpuCores, int pipelineExecutorSize) {
+        boolean isChanged = false;
+
+        if (this.cpuCores != cpuCores) {
+            this.cpuCores = cpuCores;
+            isChanged = true;
+        }
+        if (this.pipelineExecutorSize != pipelineExecutorSize) {
+            this.pipelineExecutorSize = pipelineExecutorSize;
+            isChanged = true;
+        }
+        if (isChanged) {
+            LOG.info("add new cpu info. backendId: {}, cpucores: {}, pipelineExecutorSize: {}", id, cpuCores,
+                    pipelineExecutorSize);
+            Env.getCurrentEnv().getEditLog().logBackendStateChange(this);

Review Comment:
   不要在这加这个逻辑



##########
fe/fe-core/src/main/java/org/apache/doris/system/Backend.java:
##########
@@ -519,6 +542,24 @@ public void updateDisks(Map<String, TDisk> backendDisks) {
         }
     }
 
+    public void updateCpuInfo(int cpuCores, int pipelineExecutorSize) {
+        boolean isChanged = false;
+
+        if (this.cpuCores != cpuCores) {
+            this.cpuCores = cpuCores;
+            isChanged = true;
+        }
+        if (this.pipelineExecutorSize != pipelineExecutorSize) {
+            this.pipelineExecutorSize = pipelineExecutorSize;
+            isChanged = true;
+        }
+        if (isChanged) {
+            LOG.info("add new cpu info. backendId: {}, cpucores: {}, pipelineExecutorSize: {}", id, cpuCores,
+                    pipelineExecutorSize);
+            Env.getCurrentEnv().getEditLog().logBackendStateChange(this);

Review Comment:
   不要在这加这个逻辑



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626404729

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626889720

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626889732

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei commented on a diff in pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #21640:
URL: https://github.com/apache/doris/pull/21640#discussion_r1256717725


##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -256,7 +256,7 @@ protected void exec() {
                 ReportHandler.taskReport(beId, tasks);
             }
             if (disks != null) {

Review Comment:
   不要使用disk report了,单独写一个 cpuInfoReport



##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -256,7 +256,7 @@ protected void exec() {
                 ReportHandler.taskReport(beId, tasks);
             }
             if (disks != null) {

Review Comment:
   不要使用disk report了,单独写一个 cpuInfoReport



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] Mryange commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626626306

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] Mryange commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626402836

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] BiteTheDDDDt merged pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #21640:
URL: https://github.com/apache/doris/pull/21640


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #21640:
URL: https://github.com/apache/doris/pull/21640#discussion_r1257273721


##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1433,8 +1433,8 @@ public void setEnableFoldConstantByBe(boolean foldConstantByBe) {
 
     public int getParallelExecInstanceNum() {
         if (enablePipelineEngine && parallelPipelineTaskNum == 0) {
-            Backend.BeInfoCollector beinfoCollector = Backend.getBeInfoCollector();
-            return beinfoCollector.getParallelExecInstanceNum();
+            int size = Env.getCurrentSystemInfo().getMinPipelineExecutorSize();
+            return (size + 1) / 2;

Review Comment:
   Is it possible that size is int_max?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] github-actions[bot] commented on pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21640:
URL: https://github.com/apache/doris/pull/21640#issuecomment-1626648415

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [doris] yiguolei commented on a diff in pull request #21640: [refactor](planner) refactor automatically set instance_num

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #21640:
URL: https://github.com/apache/doris/pull/21640#discussion_r1256718411


##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -549,16 +549,17 @@ private static void taskReport(long backendId, Map<TTaskType, Set<Long>> running
                 backendId, batchTask.getTaskNum(), (System.currentTimeMillis() - start));
     }
 
-    private static void diskReport(long backendId, Map<String, TDisk> backendDisks) {
+    private static void diskReport(long backendId, Map<String, TDisk> backendDisks, int cpuCores,
+            int pipelineExecutorSize) {
         LOG.info("begin to handle disk report from backend {}", backendId);
         long start = System.currentTimeMillis();
         Backend backend = Env.getCurrentSystemInfo().getBackend(backendId);
         if (backend == null) {
             LOG.warn("backend doesn't exist. id: " + backendId);
             return;
         }
-
         backend.updateDisks(backendDisks);
+        backend.updateCpuInfo(cpuCores, pipelineExecutorSize);

Review Comment:
   在这里判断cpu或者pipeline 配置是否有变更,如果变更了,写system info



##########
fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java:
##########
@@ -549,16 +549,17 @@ private static void taskReport(long backendId, Map<TTaskType, Set<Long>> running
                 backendId, batchTask.getTaskNum(), (System.currentTimeMillis() - start));
     }
 
-    private static void diskReport(long backendId, Map<String, TDisk> backendDisks) {
+    private static void diskReport(long backendId, Map<String, TDisk> backendDisks, int cpuCores,
+            int pipelineExecutorSize) {
         LOG.info("begin to handle disk report from backend {}", backendId);
         long start = System.currentTimeMillis();
         Backend backend = Env.getCurrentSystemInfo().getBackend(backendId);
         if (backend == null) {
             LOG.warn("backend doesn't exist. id: " + backendId);
             return;
         }
-
         backend.updateDisks(backendDisks);
+        backend.updateCpuInfo(cpuCores, pipelineExecutorSize);

Review Comment:
   在这里判断cpu或者pipeline 配置是否有变更,如果变更了,写system info



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org