You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2023/10/29 16:54:20 UTC

[PR] [feature](diagnose) diagnose for cluster balance [doris]

yujun777 opened a new pull request, #26085:
URL: https://github.com/apache/doris/pull/26085

   ## Proposed changes
   
   Usage: `show proc "/diagnose/cluster_balance"`
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784318407

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784332295

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1788329812

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.28 seconds
    stream load tsv:          550 seconds loaded 74807831229 Bytes, about 129 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
    storage size: 17162271978 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1788292110

   run clickbench


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1793630318

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1803020990

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1795074890

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784347604

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.08 seconds
    stream load tsv:          555 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17161892593 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #26085:
URL: https://github.com/apache/doris/pull/26085


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1795074745

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784175952

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 48.82 seconds
    stream load tsv:          557 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.9 seconds inserted 10000000 Rows, about 334K ops/s
    storage size: 17162036180 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1803055062

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.47 seconds
    stream load tsv:          561 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17162192501 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1805200374

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1793635275

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.09 seconds
    stream load tsv:          554 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17162476483 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "deardeng (via GitHub)" <gi...@apache.org>.
deardeng commented on code in PR #26085:
URL: https://github.com/apache/doris/pull/26085#discussion_r1377658249


##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/DiagnoseClusterBalanceProcDir.java:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.proc;
+
+import org.apache.doris.catalog.ColocateTableIndex.GroupId;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TabletInvertedIndex;
+import org.apache.doris.clone.BackendLoadStatistic;
+import org.apache.doris.clone.BackendLoadStatistic.Classification;
+import org.apache.doris.clone.LoadStatisticForTag;
+import org.apache.doris.clone.RootPathLoadStatistic;
+import org.apache.doris.clone.SchedException.SubCode;
+import org.apache.doris.clone.TabletSchedCtx;
+import org.apache.doris.clone.TabletScheduler;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.proc.DiagnoseProcDir.DiagnoseItem;
+import org.apache.doris.common.proc.DiagnoseProcDir.DiagnoseStatus;
+import org.apache.doris.common.proc.DiagnoseProcDir.SubProcDir;
+import org.apache.doris.common.proc.TabletHealthProcDir.DBTabletStatistic;
+import org.apache.doris.ha.FrontendNodeType;
+import org.apache.doris.resource.Tag;
+import org.apache.doris.system.SystemInfoService;
+import org.apache.doris.thrift.TStorageMedium;
+
+import com.google.common.collect.Lists;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/*
+ * show proc "/diagnose/cluster_balance";
+ */
+public class DiagnoseClusterBalanceProcDir extends SubProcDir {
+
+    @Override
+    public List<DiagnoseItem> getDiagnoseResult() {
+        long now = System.currentTimeMillis();
+        long minToMs = 60 * 1000L;
+
+        Env env = Env.getCurrentEnv();
+        SystemInfoService infoService = env.getClusterInfo();
+        TabletScheduler tabletScheduler = env.getTabletScheduler();
+        List<TabletSchedCtx> pendingTablets = tabletScheduler.getPendingTablets(1);
+        List<TabletSchedCtx> runningTablets = tabletScheduler.getRunningTablets(1);
+        List<TabletSchedCtx> historyTablets = tabletScheduler.getHistoryTablets(10000);
+        Map<Tag, LoadStatisticForTag> loadStatisticMap = tabletScheduler.getStatisticMap();
+        long historyLastVisitTime = historyTablets.stream()
+                .mapToLong(tablet -> Math.max(tablet.getCreateTime(), tablet.getLastVisitedTime()))
+                .max().orElse(-1);
+        boolean schedReady = env.getFrontends(null).stream().anyMatch(
+                fe -> fe.isAlive() && now >= fe.getLastStartupTime() + 1 * minToMs
+                        && (fe.getRole() == FrontendNodeType.MASTER || fe.getRole() == FrontendNodeType.FOLLOWER));
+        boolean schedRecent = !pendingTablets.isEmpty() || !runningTablets.isEmpty()
+                || historyLastVisitTime >= now - 15 * minToMs;
+
+        List<DiagnoseItem> items = Lists.newArrayList();
+        items.add(diagnoseTabletHealth(schedReady, schedRecent));
+        DiagnoseItem baseBalance = diagnoseBaseBalance(infoService, loadStatisticMap, schedReady, schedRecent);
+        items.add(baseBalance);
+        items.add(diagnoseDiskBalance(infoService, loadStatisticMap, schedReady, schedRecent,
+                baseBalance.status == DiagnoseStatus.OK));
+        items.add(diagnoseColocateRebalance(schedReady, schedRecent));
+        items.add(diagnoseHistorySched(historyTablets,
+                items.stream().allMatch(item -> item.status == DiagnoseStatus.OK)));
+
+        return items;
+    }
+
+    private DiagnoseItem diagnoseTabletHealth(boolean schedReady, boolean schedRecent) {
+        DiagnoseItem tabletHealth = new DiagnoseItem();
+        tabletHealth.name = "Tablet Health";
+        tabletHealth.status = DiagnoseStatus.OK;
+
+        Env env = Env.getCurrentEnv();
+        List<DBTabletStatistic> statistics = env.getInternalCatalog().getDbIds().parallelStream()
+                // skip information_schema database
+                .flatMap(id -> Stream.of(id == 0 ? null : env.getInternalCatalog().getDbNullable(id)))
+                .filter(Objects::nonNull).map(DBTabletStatistic::new)
+                // sort by dbName
+                .sorted(Comparator.comparing(db -> db.db.getFullName())).collect(Collectors.toList());
+
+        DBTabletStatistic total = statistics.stream().reduce(new DBTabletStatistic(), DBTabletStatistic::reduce);
+        if (total.tabletNum != total.healthyNum) {
+            tabletHealth.status = DiagnoseStatus.ERROR;
+            tabletHealth.content = String.format("healthy tablet num %s < total tablet num %s",
+                    total.healthyNum, total.tabletNum);
+            tabletHealth.detailCmd = "show proc \"/cluster_health/tablet_health\";";
+            boolean changeWarning = total.unrecoverableNum == 0;
+            if (Config.disable_tablet_scheduler) {
+                tabletHealth.suggestion = "has disable tablet balance, ensure master fe config: "
+                        + "disable_tablet_scheduler = false";
+            } else if (!schedReady) {
+                tabletHealth.suggestion = "check all fe are ready, and then wait some minutes for "
+                        + "sheduler to migrate tablets";
+            } else if (schedRecent) {
+                tabletHealth.suggestion = "tablet is still scheduling, run 'show proc \"/cluster_balance\"'";
+            } else {
+                changeWarning = false;
+            }
+            if (changeWarning) {
+                tabletHealth.status = DiagnoseStatus.WARNING;
+            }
+        }
+
+        return tabletHealth;
+    }
+
+    private DiagnoseItem diagnoseBaseBalance(SystemInfoService infoService,
+            Map<Tag, LoadStatisticForTag> loadStatisticMap, boolean schedReady, boolean schedRecent) {
+        DiagnoseItem baseBalance = new DiagnoseItem();
+        baseBalance.status = DiagnoseStatus.OK;
+
+        // check base balance
+        List<Long> availableBeIds = infoService.getAllBackendIds(true).stream()
+                .filter(beId -> infoService.checkBackendScheduleAvailable(beId))
+                .collect(Collectors.toList());
+        boolean isPartitionBal = Config.tablet_rebalancer_type.equalsIgnoreCase("partition");
+        if (isPartitionBal) {
+            TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
+            baseBalance.name = "Partition Balance";
+            List<Integer> tabletNums = availableBeIds.stream()
+                    .map(beId -> invertedIndex.getTabletNumByBackendId(beId))
+                    .collect(Collectors.toList());
+            int minTabletNum = tabletNums.stream().mapToInt(v -> v).min().orElse(0);
+            int maxTabletNum = tabletNums.stream().mapToInt(v -> v).max().orElse(0);
+            if (maxTabletNum <= Math.max(minTabletNum * 1.1, minTabletNum + 50)) {

Review Comment:
   1.1、50 add some comment or use config num?



##########
fe/fe-core/src/main/java/org/apache/doris/common/proc/DiagnoseClusterBalanceProcDir.java:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.proc;
+
+import org.apache.doris.catalog.ColocateTableIndex.GroupId;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TabletInvertedIndex;
+import org.apache.doris.clone.BackendLoadStatistic;
+import org.apache.doris.clone.BackendLoadStatistic.Classification;
+import org.apache.doris.clone.LoadStatisticForTag;
+import org.apache.doris.clone.RootPathLoadStatistic;
+import org.apache.doris.clone.SchedException.SubCode;
+import org.apache.doris.clone.TabletSchedCtx;
+import org.apache.doris.clone.TabletScheduler;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.proc.DiagnoseProcDir.DiagnoseItem;
+import org.apache.doris.common.proc.DiagnoseProcDir.DiagnoseStatus;
+import org.apache.doris.common.proc.DiagnoseProcDir.SubProcDir;
+import org.apache.doris.common.proc.TabletHealthProcDir.DBTabletStatistic;
+import org.apache.doris.ha.FrontendNodeType;
+import org.apache.doris.resource.Tag;
+import org.apache.doris.system.SystemInfoService;
+import org.apache.doris.thrift.TStorageMedium;
+
+import com.google.common.collect.Lists;
+
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/*
+ * show proc "/diagnose/cluster_balance";
+ */
+public class DiagnoseClusterBalanceProcDir extends SubProcDir {
+
+    @Override
+    public List<DiagnoseItem> getDiagnoseResult() {
+        long now = System.currentTimeMillis();
+        long minToMs = 60 * 1000L;
+
+        Env env = Env.getCurrentEnv();
+        SystemInfoService infoService = env.getClusterInfo();
+        TabletScheduler tabletScheduler = env.getTabletScheduler();
+        List<TabletSchedCtx> pendingTablets = tabletScheduler.getPendingTablets(1);
+        List<TabletSchedCtx> runningTablets = tabletScheduler.getRunningTablets(1);
+        List<TabletSchedCtx> historyTablets = tabletScheduler.getHistoryTablets(10000);
+        Map<Tag, LoadStatisticForTag> loadStatisticMap = tabletScheduler.getStatisticMap();
+        long historyLastVisitTime = historyTablets.stream()
+                .mapToLong(tablet -> Math.max(tablet.getCreateTime(), tablet.getLastVisitedTime()))
+                .max().orElse(-1);
+        boolean schedReady = env.getFrontends(null).stream().anyMatch(
+                fe -> fe.isAlive() && now >= fe.getLastStartupTime() + 1 * minToMs
+                        && (fe.getRole() == FrontendNodeType.MASTER || fe.getRole() == FrontendNodeType.FOLLOWER));
+        boolean schedRecent = !pendingTablets.isEmpty() || !runningTablets.isEmpty()
+                || historyLastVisitTime >= now - 15 * minToMs;
+
+        List<DiagnoseItem> items = Lists.newArrayList();
+        items.add(diagnoseTabletHealth(schedReady, schedRecent));
+        DiagnoseItem baseBalance = diagnoseBaseBalance(infoService, loadStatisticMap, schedReady, schedRecent);
+        items.add(baseBalance);
+        items.add(diagnoseDiskBalance(infoService, loadStatisticMap, schedReady, schedRecent,
+                baseBalance.status == DiagnoseStatus.OK));
+        items.add(diagnoseColocateRebalance(schedReady, schedRecent));
+        items.add(diagnoseHistorySched(historyTablets,
+                items.stream().allMatch(item -> item.status == DiagnoseStatus.OK)));
+
+        return items;
+    }
+
+    private DiagnoseItem diagnoseTabletHealth(boolean schedReady, boolean schedRecent) {
+        DiagnoseItem tabletHealth = new DiagnoseItem();
+        tabletHealth.name = "Tablet Health";
+        tabletHealth.status = DiagnoseStatus.OK;
+
+        Env env = Env.getCurrentEnv();
+        List<DBTabletStatistic> statistics = env.getInternalCatalog().getDbIds().parallelStream()
+                // skip information_schema database
+                .flatMap(id -> Stream.of(id == 0 ? null : env.getInternalCatalog().getDbNullable(id)))
+                .filter(Objects::nonNull).map(DBTabletStatistic::new)
+                // sort by dbName
+                .sorted(Comparator.comparing(db -> db.db.getFullName())).collect(Collectors.toList());
+
+        DBTabletStatistic total = statistics.stream().reduce(new DBTabletStatistic(), DBTabletStatistic::reduce);
+        if (total.tabletNum != total.healthyNum) {
+            tabletHealth.status = DiagnoseStatus.ERROR;
+            tabletHealth.content = String.format("healthy tablet num %s < total tablet num %s",
+                    total.healthyNum, total.tabletNum);
+            tabletHealth.detailCmd = "show proc \"/cluster_health/tablet_health\";";
+            boolean changeWarning = total.unrecoverableNum == 0;
+            if (Config.disable_tablet_scheduler) {
+                tabletHealth.suggestion = "has disable tablet balance, ensure master fe config: "
+                        + "disable_tablet_scheduler = false";
+            } else if (!schedReady) {
+                tabletHealth.suggestion = "check all fe are ready, and then wait some minutes for "
+                        + "sheduler to migrate tablets";
+            } else if (schedRecent) {
+                tabletHealth.suggestion = "tablet is still scheduling, run 'show proc \"/cluster_balance\"'";
+            } else {
+                changeWarning = false;
+            }
+            if (changeWarning) {
+                tabletHealth.status = DiagnoseStatus.WARNING;
+            }
+        }
+
+        return tabletHealth;
+    }
+
+    private DiagnoseItem diagnoseBaseBalance(SystemInfoService infoService,
+            Map<Tag, LoadStatisticForTag> loadStatisticMap, boolean schedReady, boolean schedRecent) {
+        DiagnoseItem baseBalance = new DiagnoseItem();
+        baseBalance.status = DiagnoseStatus.OK;
+
+        // check base balance
+        List<Long> availableBeIds = infoService.getAllBackendIds(true).stream()
+                .filter(beId -> infoService.checkBackendScheduleAvailable(beId))
+                .collect(Collectors.toList());
+        boolean isPartitionBal = Config.tablet_rebalancer_type.equalsIgnoreCase("partition");
+        if (isPartitionBal) {
+            TabletInvertedIndex invertedIndex = Env.getCurrentInvertedIndex();
+            baseBalance.name = "Partition Balance";
+            List<Integer> tabletNums = availableBeIds.stream()
+                    .map(beId -> invertedIndex.getTabletNumByBackendId(beId))
+                    .collect(Collectors.toList());
+            int minTabletNum = tabletNums.stream().mapToInt(v -> v).min().orElse(0);
+            int maxTabletNum = tabletNums.stream().mapToInt(v -> v).max().orElse(0);
+            if (maxTabletNum <= Math.max(minTabletNum * 1.1, minTabletNum + 50)) {

Review Comment:
   1.1、50 add some comment or use config num?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1803079668

   
   <details>
   <summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
   
   ```
   Tpch sf100 test result on commit bf6382273f1cb812d296a9aea2c720d974bdb2e7, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	5227	5108	5130	5108
   q2	389	213	289	213
   q3	2081	2006	2019	2006
   q4	1455	1420	1403	1403
   q5	4106	4164	4080	4080
   q6	247	133	126	126
   q7	2045	1600	1616	1600
   q8	2919	2723	2722	2722
   q9	10592	10255	10219	10219
   q10	3477	3575	3562	3562
   q11	370	259	245	245
   q12	443	305	296	296
   q13	4504	4126	4134	4126
   q14	320	300	287	287
   q15	658	569	569	569
   q16	688	608	603	603
   q17	1117	1098	1071	1071
   q18	8020	7325	7357	7325
   q19	1668	1683	1688	1683
   q20	568	383	356	356
   q21	4915	4484	4546	4484
   q22	534	430	429	429
   Total cold run time: 56343 ms
   Total hot run time: 52513 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4966	5084	4956	4956
   q2	350	247	241	241
   q3	4033	4014	4019	4014
   q4	2807	2771	2762	2762
   q5	6453	6411	6415	6411
   q6	239	125	127	125
   q7	3121	2740	2674	2674
   q8	4686	4717	4665	4665
   q9	17769	17656	17652	17652
   q10	4101	4205	4155	4155
   q11	732	657	638	638
   q12	984	824	888	824
   q13	4271	3881	3903	3881
   q14	367	345	334	334
   q15	640	567	570	567
   q16	786	670	696	670
   q17	3895	3885	3867	3867
   q18	9550	9271	9306	9271
   q19	1816	1734	1752	1734
   q20	2367	2051	2043	2043
   q21	8793	8672	8746	8672
   q22	956	848	832	832
   Total cold run time: 83682 ms
   Total hot run time: 80988 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784166012

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [feature](diagnose) diagnose for cluster balance [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26085:
URL: https://github.com/apache/doris/pull/26085#issuecomment-1784363192

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.43 seconds
    stream load tsv:          556 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
    storage size: 17162160073 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org