You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangbo (via GitHub)" <gi...@apache.org> on 2024/07/04 08:10:07 UTC
[PR] [Improvement]Log be thread num [doris]
wangbo opened a new pull request, #37289:
URL: https://github.com/apache/doris/pull/37289
## Proposed changes
Log doris_be process's thread num at a fixed interval, when a BE cores because of thread exhaustion, we can use this log to find which logic uses the most threads.
```
log be thread num: pid=3545627 thread_name_count=106 total_thread_num=7103
details: RScan_auth_grou=960
RScan_normal [w=960
RScan_test_wg [=960
RemoteScanThrea=960
brpc_heavy=384
brpc_light=384
doris_be=294
Scan_normal [wo=192
Scan_auth_group=192
LimitedScanThre=192
Scan_test_wg [w=192
SpillIOThreadPo=192
local_scan=192
Pipe_test_wg [w=96
Pipe_normal [wo=96
Pipe_auth_group=96
PipeNoGSchePool=96
DownloadThreadP=64
SendReportThrea=64
SendBatchThread=64
FragmentMgrThre=64
JoinNodeThreadP=64
EvHttpServer [w=48
FragmentInstanc=48
TabletPublishTx=32
S3FileUploadThr=16
BufferedReaderP=16
NonBlockCloseTh=12
MultiGetTaskThr=10
TaskWP_PUBLISH_=8
SendTableStatsT=8
MemTableHighPri=6
MemTableFlushTh=6
wg_flush_test_w=6
wg_flush_auth_g=6
SegmentFileWrit=6
wg_flush_normal=6
WriteCooldownMe=5
TaskWP_MAKE_SNA=5
CooldownTaskThr=5
SegCompactionTa=5
TaskWP_RELEASE_=5
BaseCompactionT=4
cancel_timeout_=3
TaskWP_DELETE [=3
TaskWP_DROP_TAB=3
TaskWP_CREATE_T=3
TaskWP_CLONE [w=3
TaskWP_ALTER_TA=3
TaskWP_ALTER_IN=3
TaskWP_.PUSH [w=3
HighPriorPool.P=3
ColdDataCompact=2
memtable_memory=1
query_runtime_s=1
memory_maintena=1
memory_gc_threa=1
GroupCommitThre=1
je_purge_dirty_=1
gc_expired_cont=1
garbage_sweeper=1
evict_querying_=1
disk_stat_monit=1
cooldown_tasks_=1
compaction_task=1
cold_data_compa=1
path_gc_thread-=1
ping_worker-354=1
clean_expired_t=1
remove_unused_r=1
replay_wal-3547=1
spill_gc_thread=1
tablet_checkpoi=1
unused_rowset_m=1
update_wal_dir_=1
GroupCommitRepl=1
CumuCompactionT=1
wg_mem_refresh_=1
workload_schedu=1
TaskWP_UPDATE_T=1
TaskWP_CLEAN_UD=1
TaskWP_CLEAR_TR=1
TaskWP_CHECK_CO=1
TaskWP_DOWNLOAD=1
TaskWP_GC_BINLO=1
TabletCalcDelet=1
TaskWP_MOVE [wo=1
SingleReplicaCo=1
TaskWP_PUSH_COO=1
TaskWP_PUSH_STO=1
REPORT_TASK-354=1
TaskWP_STORAGE_=1
TaskWP_SUBMIT_T=1
clean_idle_cons=1
TaskWP_UPDATE_V=1
TaskWP_UPLOAD [=1
REPORT_OLAP_TAB=1
_update_replica=1
async_publish_v=1
be_proc_monitor=1
REPORT_DISK_STA=1
MigrationTaskTh=1
cache_clean_thr=1
calculate_metri=1
LazyReleaseMemo=1
TaskWP_CLEAN_TR=1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208698876
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208584312
<details>
<summary>ClickBench: <b>Total hot run time: 30.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 672c824a4377d8f41ab19323b8161d9a71506782, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.03 0.04
query3 0.22 0.05 0.05
query4 1.68 0.07 0.10
query5 0.50 0.48 0.49
query6 1.15 0.73 0.73
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.55 0.48 0.49
query10 0.55 0.55 0.54
query11 0.14 0.11 0.12
query12 0.14 0.13 0.12
query13 0.59 0.59 0.58
query14 0.77 0.78 0.78
query15 0.85 0.81 0.82
query16 0.35 0.35 0.35
query17 0.98 1.03 1.03
query18 0.22 0.26 0.25
query19 1.77 1.76 1.72
query20 0.02 0.01 0.01
query21 15.40 0.76 0.65
query22 4.53 7.27 1.79
query23 18.27 1.41 1.44
query24 2.13 0.22 0.22
query25 0.16 0.08 0.09
query26 0.26 0.16 0.17
query27 0.08 0.09 0.08
query28 13.19 1.02 1.01
query29 12.65 3.43 3.46
query30 0.25 0.06 0.07
query31 2.86 0.39 0.39
query32 3.28 0.49 0.46
query33 2.88 2.91 2.90
query34 16.97 4.40 4.46
query35 4.54 4.50 4.53
query36 0.65 0.46 0.46
query37 0.18 0.16 0.15
query38 0.15 0.15 0.14
query39 0.04 0.03 0.04
query40 0.18 0.14 0.14
query41 0.10 0.04 0.04
query42 0.05 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.51 s
Total hot run time: 30.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo merged PR #37289:
URL: https://github.com/apache/doris/pull/37289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208698954
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665419849
##########
be/src/common/daemon.cpp:
##########
@@ -396,6 +397,14 @@ void Daemon::wg_mem_used_refresh_thread() {
}
}
+void Daemon::be_proc_monitor_thread() {
+ while (!_stop_background_threads_latch.wait_for(
+ std::chrono::milliseconds(config::be_proc_monitor_interval_ms))) {
+ BeProcMonitor bpm;
Review Comment:
should we create object each while loop? Better the `get_be_thread_info()` should be static
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1666341619
##########
be/src/http/action/be_proc_thread_action.h:
##########
@@ -0,0 +1,33 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+
+#include "http/http_handler.h"
+#include "http/http_request.h"
+
+namespace doris {
+
+class HttpRequest;
+
+class BeProcThreadAction : public HttpHandler {
+public:
+ BeProcThreadAction() = default;
+ ~BeProcThreadAction() override = default;
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
```suggestion
~BeProcThreadAction() override = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665450176
##########
be/src/runtime/be_proc_monitor.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/be_proc_monitor.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <filesystem>
+#include <fstream>
+#include <map>
+
+std::string BeProcMonitor::get_be_thread_info() {
+ int32_t pid = getpid();
+ std::string proc_path = fmt::format("/proc/{}/task", pid);
+ if (access(proc_path.c_str(), F_OK) != 0) {
+ LOG(INFO) << "be proc path " << proc_path << " not exists.";
+ return "";
+ }
+
+ std::map<std::string, int> thread_num_map;
+
+ for (const auto& entry : std::filesystem::directory_iterator(proc_path)) {
+ const std::string tid_path = entry.path().string();
+ std::string thread_name_path = tid_path + "/comm";
+ struct stat st;
+ // == 0 means exists
+ if (stat(thread_name_path.c_str(), &st) == 0) {
+ // NOTE: there is no need to close std::ifstream, it's called during deconstruction.
+ // refer:https://stackoverflow.com/questions/748014/do-i-need-to-manually-close-an-ifstream
+ std::ifstream file(thread_name_path.c_str());
+ if (!file.is_open()) {
+ continue;
+ }
+ std::stringstream str_buf;
+ str_buf << file.rdbuf();
+ std::string thread_name = str_buf.str();
+ thread_name.erase(std::remove(thread_name.begin(), thread_name.end(), '\n'),
+ thread_name.end());
+
+ if (thread_num_map.find(thread_name) != thread_num_map.end()) {
+ int count = thread_num_map.at(thread_name);
+ count++;
+ thread_num_map[thread_name] = count;
Review Comment:
thread_num_map[thread_name]++;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665421681
##########
be/src/runtime/be_proc_monitor.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/be_proc_monitor.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <filesystem>
+#include <fstream>
+#include <map>
+
+std::string BeProcMonitor::get_be_thread_info() {
Review Comment:
why direct log in the function ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2210229370
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665419849
##########
be/src/common/daemon.cpp:
##########
@@ -396,6 +397,14 @@ void Daemon::wg_mem_used_refresh_thread() {
}
}
+void Daemon::be_proc_monitor_thread() {
+ while (!_stop_background_threads_latch.wait_for(
+ std::chrono::milliseconds(config::be_proc_monitor_interval_ms))) {
+ BeProcMonitor bpm;
Review Comment:
should we create a object each while loop，better the `get_be_thread_info()` should be static
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208573594
<details>
<summary>TPC-DS: <b>Total hot run time: 172444 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 672c824a4377d8f41ab19323b8161d9a71506782, data reload: false
query1 938 385 377 377
query2 6463 2461 2438 2438
query3 6633 209 218 209
query4 19039 17657 17405 17405
query5 3657 506 477 477
query6 268 170 189 170
query7 4597 290 304 290
query8 322 297 295 295
query9 8586 2390 2375 2375
query10 580 308 295 295
query11 10539 10028 10259 10028
query12 112 87 85 85
query13 1642 381 379 379
query14 9498 7143 6229 6229
query15 219 179 182 179
query16 7752 317 298 298
query17 1701 562 519 519
query18 1956 274 267 267
query19 197 153 150 150
query20 90 81 80 80
query21 212 130 124 124
query22 4351 4072 4180 4072
query23 33983 33551 33382 33382
query24 10777 2971 2802 2802
query25 599 377 376 376
query26 713 156 152 152
query27 2254 324 334 324
query28 6141 2187 2165 2165
query29 888 637 620 620
query30 270 164 166 164
query31 983 749 756 749
query32 96 56 53 53
query33 675 298 297 297
query34 864 471 491 471
query35 750 650 631 631
query36 1144 995 978 978
query37 146 85 83 83
query38 2925 2872 2846 2846
query39 900 819 801 801
query40 216 133 128 128
query41 54 53 51 51
query42 122 104 105 104
query43 581 537 558 537
query44 1071 770 744 744
query45 192 173 169 169
query46 1067 738 710 710
query47 1863 1767 1781 1767
query48 362 306 308 306
query49 825 410 421 410
query50 761 383 390 383
query51 6917 6657 6793 6657
query52 112 92 93 92
query53 352 290 286 286
query54 881 448 449 448
query55 75 72 76 72
query56 284 283 270 270
query57 1113 1050 1019 1019
query58 255 250 253 250
query59 3509 3205 3272 3205
query60 303 279 287 279
query61 96 92 89 89
query62 604 453 427 427
query63 317 287 299 287
query64 8463 2229 1900 1900
query65 3191 3102 3096 3096
query66 766 387 321 321
query67 15427 15311 14998 14998
query68 4578 560 564 560
query69 681 434 324 324
query70 1170 1095 1118 1095
query71 406 312 279 279
query72 7806 5097 5276 5097
query73 748 337 324 324
query74 5856 5587 5594 5587
query75 3496 2667 2691 2667
query76 2649 954 963 954
query77 617 308 309 308
query78 9420 9212 8838 8838
query79 3868 531 520 520
query80 1763 504 481 481
query81 539 218 217 217
query82 874 104 110 104
query83 337 179 181 179
query84 270 86 100 86
query85 1750 281 335 281
query86 469 316 310 310
query87 3275 3082 3087 3082
query88 4562 2461 2466 2461
query89 491 383 391 383
query90 1777 185 186 185
query91 130 103 99 99
query92 65 50 50 50
query93 5492 513 521 513
query94 1178 213 205 205
query95 408 309 305 305
query96 588 274 271 271
query97 3171 3061 3008 3008
query98 227 203 200 200
query99 1154 839 831 831
Total cold run time: 273054 ms
Total hot run time: 172444 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2210358582
<details>
<summary>ClickBench: <b>Total hot run time: 31.19 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24323711f20c9b33c1a58ef36b6a77fb11af1ef2, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.03 0.03
query3 0.22 0.06 0.05
query4 1.66 0.09 0.10
query5 0.49 0.49 0.48
query6 1.15 0.73 0.73
query7 0.02 0.01 0.02
query8 0.05 0.05 0.04
query9 0.55 0.49 0.49
query10 0.55 0.54 0.54
query11 0.16 0.11 0.12
query12 0.14 0.12 0.12
query13 0.58 0.58 0.58
query14 0.77 0.77 0.77
query15 0.84 0.82 0.80
query16 0.37 0.37 0.37
query17 1.04 1.04 1.03
query18 0.21 0.24 0.26
query19 1.87 1.72 1.86
query20 0.01 0.01 0.01
query21 15.40 0.75 0.64
query22 3.73 6.55 2.39
query23 18.27 1.36 1.28
query24 2.12 0.23 0.22
query25 0.15 0.09 0.09
query26 0.30 0.20 0.20
query27 0.45 0.23 0.22
query28 13.29 1.01 1.00
query29 12.58 3.35 3.35
query30 0.26 0.06 0.07
query31 2.85 0.38 0.39
query32 3.29 0.46 0.47
query33 2.83 2.93 2.91
query34 17.00 4.39 4.37
query35 4.41 4.36 4.42
query36 0.65 0.46 0.47
query37 0.18 0.16 0.15
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.15 0.13 0.12
query41 0.09 0.04 0.04
query42 0.05 0.05 0.04
query43 0.05 0.04 0.05
Total cold run time: 109.09 s
Total hot run time: 31.19 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2213698893
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2210335387
<details>
<summary>TPC-H: <b>Total hot run time: 40032 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24323711f20c9b33c1a58ef36b6a77fb11af1ef2, data reload: false
------ Round 1 ----------------------------------
q1 17638 4417 4325 4325
q2 2022 191 183 183
q3 10464 1208 1055 1055
q4 10203 850 822 822
q5 7503 2762 2631 2631
q6 225 135 135 135
q7 971 587 600 587
q8 9230 2093 2062 2062
q9 8984 6512 6507 6507
q10 9016 3724 3728 3724
q11 470 241 237 237
q12 428 228 225 225
q13 17771 2954 2988 2954
q14 275 227 236 227
q15 532 477 482 477
q16 511 380 376 376
q17 981 613 697 613
q18 8125 7586 7505 7505
q19 5434 1463 1554 1463
q20 674 326 348 326
q21 4957 3257 4039 3257
q22 401 341 348 341
Total cold run time: 116815 ms
Total hot run time: 40032 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4445 4208 4213 4208
q2 367 256 254 254
q3 2999 2883 2887 2883
q4 1997 1659 1747 1659
q5 5608 5508 5432 5432
q6 226 130 129 129
q7 2224 1884 1903 1884
q8 3266 3411 3443 3411
q9 8671 8723 8807 8723
q10 4063 3900 3806 3806
q11 596 534 514 514
q12 830 644 665 644
q13 17031 3163 3154 3154
q14 298 287 279 279
q15 537 497 481 481
q16 511 448 436 436
q17 1826 1530 1517 1517
q18 8105 7811 8067 7811
q19 1782 1679 1571 1571
q20 2202 1871 1821 1821
q21 5138 4789 4973 4789
q22 637 535 566 535
Total cold run time: 73359 ms
Total hot run time: 55941 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665321664
##########
be/src/runtime/be_proc_monitor.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/be_proc_monitor.h"
+
+#include <fmt/format.h>
Review Comment:
warning: 'fmt/format.h' file not found [clang-diagnostic-error]
```cpp
#include <fmt/format.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208373011
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665423977
##########
be/src/runtime/be_proc_monitor.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/be_proc_monitor.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <filesystem>
+#include <fstream>
+#include <map>
+
+std::string BeProcMonitor::get_be_thread_info() {
+ int32_t pid = getpid();
+ std::string proc_path = fmt::format("/proc/{}/task", pid);
+ if (access(proc_path.c_str(), F_OK) != 0) {
+ LOG(INFO) << "be proc path " << proc_path << " not exists.";
+ return "";
+ }
+
+ std::map<std::string, int> thread_num_map;
+
+ for (const auto& entry : std::filesystem::directory_iterator(proc_path)) {
+ const std::string tid_path = entry.path().string();
+ std::string thread_name_path = tid_path + "/comm";
+ struct stat st;
+ // == 0 means exists
+ if (stat(thread_name_path.c_str(), &st) == 0) {
+ // NOTE: there is no need to close std::ifstream, it's called during deconstruction.
+ // refer:https://stackoverflow.com/questions/748014/do-i-need-to-manually-close-an-ifstream
+ std::ifstream file(thread_name_path.c_str());
+ if (!file.is_open()) {
+ continue;
+ }
+ std::stringstream str_buf;
+ str_buf << file.rdbuf();
+ std::string thread_name = str_buf.str();
+ thread_name.erase(std::remove(thread_name.begin(), thread_name.end(), '\n'),
+ thread_name.end());
+
+ if (thread_num_map.find(thread_name) != thread_num_map.end()) {
+ int count = thread_num_map.at(thread_name);
Review Comment:
has do the find, no need to do `at`-> `[]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208369172
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208381316
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2208548952
<details>
<summary>TPC-H: <b>Total hot run time: 39630 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 672c824a4377d8f41ab19323b8161d9a71506782, data reload: false
------ Round 1 ----------------------------------
q1 17621 4395 4301 4301
q2 2040 192 184 184
q3 10470 1157 1028 1028
q4 10200 865 779 779
q5 7535 2666 2654 2654
q6 224 134 134 134
q7 949 601 608 601
q8 9234 2083 2090 2083
q9 8985 6490 6444 6444
q10 9076 3680 3741 3680
q11 453 240 237 237
q12 443 237 232 232
q13 17824 2971 3001 2971
q14 287 243 233 233
q15 522 480 485 480
q16 509 383 387 383
q17 988 648 662 648
q18 8119 7649 7364 7364
q19 6357 1425 1432 1425
q20 672 321 341 321
q21 5022 3110 3322 3110
q22 419 339 338 338
Total cold run time: 117949 ms
Total hot run time: 39630 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4378 4247 4211 4211
q2 368 279 273 273
q3 3033 2897 2896 2896
q4 1984 1788 1754 1754
q5 5624 5489 5447 5447
q6 226 136 128 128
q7 2218 1869 1823 1823
q8 3321 3405 3438 3405
q9 8737 8647 8776 8647
q10 4172 3871 3714 3714
q11 620 505 496 496
q12 831 644 642 642
q13 16073 3177 3167 3167
q14 314 283 284 283
q15 538 487 491 487
q16 498 423 437 423
q17 1832 1520 1496 1496
q18 8048 8061 7751 7751
q19 1851 1580 1670 1580
q20 2527 1893 1867 1867
q21 7809 4795 4784 4784
q22 668 601 550 550
Total cold run time: 75670 ms
Total hot run time: 55824 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1665417848
##########
be/src/runtime/be_proc_monitor.cpp:
##########
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/be_proc_monitor.h"
+
+#include <fmt/format.h>
+#include <glog/logging.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include <filesystem>
+#include <fstream>
+#include <map>
+
+std::string BeProcMonitor::get_be_thread_info() {
+ int32_t pid = getpid();
+ std::string proc_path = fmt::format("/proc/{}/task", pid);
+ if (access(proc_path.c_str(), F_OK) != 0) {
+ LOG(INFO) << "be proc path " << proc_path << " not exists.";
Review Comment:
better warning log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37289:
URL: https://github.com/apache/doris/pull/37289#discussion_r1666254566
##########
be/src/http/action/be_proc_thread_action.h:
##########
@@ -0,0 +1,33 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#pragma once
+
+#include "http/http_handler.h"
+#include "http/http_request.h"
+
+namespace doris {
+
+class HttpRequest;
+
+class BeProcThreadAction : public HttpHandler {
+public:
+ BeProcThreadAction() = default;
+ ~BeProcThreadAction() = default;
Review Comment:
warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
```suggestion
~BeProcThreadAction() override = default;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2210223117
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement]Log be thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37289:
URL: https://github.com/apache/doris/pull/37289#issuecomment-2210350800
<details>
<summary>TPC-DS: <b>Total hot run time: 174037 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24323711f20c9b33c1a58ef36b6a77fb11af1ef2, data reload: false
query1 927 372 368 368
query2 6459 2533 2463 2463
query3 6637 212 222 212
query4 28350 17561 17306 17306
query5 3786 495 491 491
query6 268 184 163 163
query7 4584 297 296 296
query8 357 305 291 291
query9 8454 2340 2326 2326
query10 574 321 295 295
query11 11937 10025 10156 10025
query12 134 92 86 86
query13 1655 384 384 384
query14 9575 7614 7889 7614
query15 231 186 186 186
query16 7784 314 318 314
query17 1671 563 543 543
query18 1956 280 287 280
query19 202 159 159 159
query20 90 88 85 85
query21 220 130 128 128
query22 4331 4161 4065 4065
query23 34172 33451 33523 33451
query24 10883 2899 2849 2849
query25 628 423 430 423
query26 723 160 159 159
query27 2252 331 336 331
query28 5926 2125 2124 2124
query29 899 655 677 655
query30 268 167 156 156
query31 971 762 805 762
query32 102 58 58 58
query33 684 350 334 334
query34 920 495 505 495
query35 774 672 653 653
query36 1135 980 989 980
query37 146 81 86 81
query38 2898 2750 2751 2750
query39 892 815 808 808
query40 204 131 132 131
query41 53 52 53 52
query42 123 98 102 98
query43 628 565 579 565
query44 1147 752 737 737
query45 202 160 169 160
query46 1094 732 730 730
query47 1852 1779 1796 1779
query48 386 305 403 305
query49 834 415 421 415
query50 751 387 386 386
query51 6918 6796 6770 6770
query52 113 92 104 92
query53 379 301 290 290
query54 869 447 452 447
query55 75 73 74 73
query56 288 264 270 264
query57 1121 1033 1040 1033
query58 255 254 261 254
query59 3657 3223 3162 3162
query60 299 271 273 271
query61 95 94 94 94
query62 611 449 435 435
query63 329 292 291 291
query64 8587 2261 1768 1768
query65 3149 3173 3141 3141
query66 715 324 328 324
query67 15761 14968 14841 14841
query68 8915 544 554 544
query69 740 464 386 386
query70 1421 1136 1046 1046
query71 532 294 292 292
query72 9041 5484 5549 5484
query73 2241 326 321 321
query74 5821 5497 5509 5497
query75 5678 2691 2669 2669
query76 5113 974 892 892
query77 816 310 302 302
query78 9735 8934 8828 8828
query79 9385 510 529 510
query80 1967 495 475 475
query81 585 227 217 217
query82 262 105 107 105
query83 343 168 173 168
query84 275 88 85 85
query85 884 309 293 293
query86 357 306 303 303
query87 3357 3107 3144 3107
query88 4709 2447 2431 2431
query89 532 386 376 376
query90 2076 192 190 190
query91 136 104 102 102
query92 61 50 53 50
query93 5882 500 498 498
query94 1340 207 209 207
query95 404 307 317 307
query96 600 270 271 270
query97 3159 2986 2972 2972
query98 206 201 203 201
query99 1301 852 863 852
Total cold run time: 302824 ms
Total hot run time: 174037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org