You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org> on 2024/02/27 05:06:31 UTC
[PR] [refactor](profile callback) refactor profile callback on BE [doris]
zhiqiang-hhhh opened a new pull request, #31448:
URL: https://github.com/apache/doris/pull/31448
callback for profile reporting is hard to read on BE.
replace callback with member function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh closed pull request #31448: [refactor](profile callback) refactor profile callback on BE
URL: https://github.com/apache/doris/pull/31448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966118372
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 15.1 seconds inserted 10000000 Rows, about 662K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966085979
<details>
<summary>TPC-H: <b>Total hot run time: 39180 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
------ Round 1 ----------------------------------
q1 18372 4327 4314 4314
q2 2512 146 138 138
q3 12371 970 961 961
q4 4947 989 970 970
q5 8057 3012 3116 3012
q6 198 127 122 122
q7 1328 802 776 776
q8 9688 2207 2221 2207
q9 7408 6526 6479 6479
q10 8208 2522 2546 2522
q11 421 200 215 200
q12 753 305 311 305
q13 17953 3205 3196 3196
q14 285 250 257 250
q15 525 508 484 484
q16 475 394 414 394
q17 980 931 912 912
q18 6571 6010 6008 6008
q19 1723 1690 1695 1690
q20 570 273 271 271
q21 6252 3716 3691 3691
q22 809 288 278 278
Total cold run time: 110406 ms
Total hot run time: 39180 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4228 4199 4222 4199
q2 324 212 210 210
q3 3018 2979 3035 2979
q4 1881 1861 1889 1861
q5 5169 5169 5163 5163
q6 223 120 115 115
q7 2167 1720 1719 1719
q8 3318 3416 3422 3416
q9 8336 8317 8291 8291
q10 6268 3758 3703 3703
q11 543 419 434 419
q12 696 523 519 519
q13 3722 3060 3056 3056
q14 284 266 258 258
q15 563 497 485 485
q16 522 457 457 457
q17 1853 1787 1812 1787
q18 7885 7517 7586 7517
q19 1836 1831 1818 1818
q20 2133 1919 1890 1890
q21 4964 4774 4788 4774
q22 506 429 450 429
Total cold run time: 60439 ms
Total hot run time: 55065 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966454628
TeamCity be ut coverage result:
Function Coverage: 35.70% (8548/23941)
Line Coverage: 27.52% (69373/252072)
Region Coverage: 26.67% (35972/134869)
Branch Coverage: 23.47% (18383/78312)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e873221470293890959c7989f764d15b44005114_e873221470293890959c7989f764d15b44005114/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on code in PR #31448:
URL: https://github.com/apache/doris/pull/31448#discussion_r1505252281
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
can not be static since we need to extend lifetime of object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966009712
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966123071
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.3 seconds inserted 10000000 Rows, about 578K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1965795014
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966117758
<details>
<summary>ClickBench: <b>Total hot run time: 32.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
query1 0.04 0.04 0.03
query2 0.05 0.03 0.02
query3 0.24 0.06 0.06
query4 1.67 0.10 0.10
query5 0.52 0.52 0.52
query6 1.29 0.70 0.68
query7 0.02 0.01 0.02
query8 0.04 0.02 0.03
query9 0.54 0.50 0.52
query10 0.56 0.56 0.57
query11 0.13 0.10 0.10
query12 0.11 0.09 0.10
query13 0.62 0.61 0.61
query14 0.78 0.80 0.82
query15 0.84 0.82 0.80
query16 0.38 0.36 0.38
query17 1.03 1.00 1.00
query18 0.27 0.25 0.24
query19 1.83 1.70 1.78
query20 0.01 0.01 0.01
query21 15.41 0.64 0.57
query22 3.16 4.17 2.46
query23 17.33 1.08 0.95
query24 2.02 0.74 0.12
query25 0.30 0.19 0.05
query26 0.19 0.17 0.16
query27 0.06 0.04 0.04
query28 12.18 0.82 0.81
query29 12.60 3.31 3.30
query30 0.58 0.53 0.55
query31 2.79 0.36 0.35
query32 3.34 0.48 0.49
query33 3.20 3.22 3.22
query34 15.36 4.74 4.74
query35 4.77 4.75 4.76
query36 1.13 1.08 1.06
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.02
query40 0.17 0.14 0.13
query41 0.07 0.02 0.02
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.8 s
Total hot run time: 32.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966487029
<details>
<summary>TPC-DS: <b>Total hot run time: 169222 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e873221470293890959c7989f764d15b44005114, data reload: false
query1 937 345 325 325
query2 6560 1858 1787 1787
query3 6702 212 203 203
query4 23054 20624 20546 20546
query5 4340 364 352 352
query6 264 165 158 158
query7 4594 300 310 300
query8 264 192 203 192
query9 8415 2286 2265 2265
query10 415 201 213 201
query11 14578 14168 14157 14157
query12 129 81 81 81
query13 1630 444 446 444
query14 8721 6483 6582 6483
query15 201 170 175 170
query16 7089 255 257 255
query17 1034 508 508 508
query18 1895 270 265 265
query19 189 151 149 149
query20 83 83 84 83
query21 198 126 121 121
query22 4695 4593 4530 4530
query23 30483 30004 29874 29874
query24 11747 3098 3140 3098
query25 658 352 348 348
query26 1808 154 163 154
query27 2980 303 305 303
query28 6937 1820 1815 1815
query29 1175 557 565 557
query30 281 138 139 138
query31 853 688 695 688
query32 93 60 56 56
query33 727 230 222 222
query34 1023 491 488 488
query35 825 734 761 734
query36 898 864 858 858
query37 166 63 63 63
query38 3078 3025 2965 2965
query39 1315 1265 1252 1252
query40 277 103 99 99
query41 37 35 38 35
query42 104 94 96 94
query43 466 428 432 428
query44 1076 717 722 717
query45 190 178 176 176
query46 1047 794 794 794
query47 1514 1349 1431 1349
query48 428 344 351 344
query49 1171 313 290 290
query50 768 382 377 377
query51 4420 4264 4288 4264
query52 105 89 97 89
query53 337 265 262 262
query54 304 225 226 225
query55 90 82 83 82
query56 220 197 197 197
query57 1002 867 878 867
query58 213 199 195 195
query59 2322 2285 2196 2196
query60 230 217 211 211
query61 85 85 83 83
query62 589 362 379 362
query63 282 255 261 255
query64 6551 2944 3131 2944
query65 3245 3216 3192 3192
query66 1429 329 313 313
query67 14491 14079 13868 13868
query68 4983 571 579 571
query69 556 377 372 372
query70 1196 1167 1136 1136
query71 369 250 245 245
query72 6231 2844 2671 2671
query73 710 318 323 318
query74 6363 6014 5953 5953
query75 3062 2438 2400 2400
query76 2847 1029 1148 1029
query77 345 238 225 225
query78 8902 8497 8555 8497
query79 1094 514 543 514
query80 681 374 351 351
query81 450 196 195 195
query82 1297 82 81 81
query83 221 122 120 120
query84 282 80 78 78
query85 1148 366 344 344
query86 321 301 304 301
query87 3295 3108 3087 3087
query88 2718 2392 2330 2330
query89 378 333 330 330
query90 1985 167 162 162
query91 158 126 125 125
query92 53 48 45 45
query93 1052 531 507 507
query94 1031 179 177 177
query95 445 337 331 331
query96 575 268 267 267
query97 4259 4180 4142 4142
query98 216 204 196 196
query99 1022 642 626 626
Total cold run time: 262272 ms
Total hot run time: 169222 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966504063
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e873221470293890959c7989f764d15b44005114 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 16.2 seconds inserted 10000000 Rows, about 617K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh closed pull request #31448: [refactor](profile callback) refactor profile callback on BE
URL: https://github.com/apache/doris/pull/31448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966401675
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966106122
<details>
<summary>TPC-DS: <b>Total hot run time: 170099 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
query1 936 338 336 336
query2 6534 1848 1882 1848
query3 6700 210 209 209
query4 23442 20645 20620 20620
query5 4285 360 358 358
query6 285 180 166 166
query7 4610 294 290 290
query8 261 191 196 191
query9 8436 2299 2294 2294
query10 413 211 209 209
query11 14518 14144 14198 14144
query12 134 81 84 81
query13 1659 430 426 426
query14 8292 6295 6420 6295
query15 197 171 183 171
query16 7094 270 248 248
query17 1033 511 502 502
query18 1904 263 263 263
query19 192 162 145 145
query20 85 83 86 83
query21 196 118 119 118
query22 4766 4851 4689 4689
query23 30537 29994 29926 29926
query24 12181 3161 3245 3161
query25 659 354 343 343
query26 1865 152 161 152
query27 3061 316 314 314
query28 7021 1846 1845 1845
query29 1187 577 556 556
query30 283 136 140 136
query31 893 695 681 681
query32 91 64 55 55
query33 712 220 220 220
query34 1099 483 502 483
query35 861 742 752 742
query36 951 852 881 852
query37 138 59 65 59
query38 3098 2992 3017 2992
query39 1311 1262 1255 1255
query40 287 102 95 95
query41 38 36 35 35
query42 104 96 96 96
query43 463 500 491 491
query44 1054 708 711 708
query45 182 181 174 174
query46 1040 788 782 782
query47 1552 1470 1496 1470
query48 433 363 345 345
query49 1166 284 299 284
query50 767 379 378 378
query51 4364 4316 4257 4257
query52 103 88 91 88
query53 342 264 265 264
query54 297 226 222 222
query55 88 80 81 80
query56 213 204 199 199
query57 953 910 900 900
query58 207 194 194 194
query59 2289 2250 2195 2195
query60 235 216 211 211
query61 85 83 87 83
query62 621 359 376 359
query63 272 255 255 255
query64 6568 2970 3058 2970
query65 3239 3223 3198 3198
query66 1443 313 319 313
query67 14450 14299 14150 14150
query68 5062 557 573 557
query69 535 381 374 374
query70 1226 1272 1237 1237
query71 340 249 249 249
query72 6237 2780 2636 2636
query73 704 320 322 320
query74 6405 6079 6052 6052
query75 3034 2425 2401 2401
query76 2834 1053 1155 1053
query77 347 237 228 228
query78 9228 8638 8516 8516
query79 1317 515 508 508
query80 951 346 339 339
query81 485 192 199 192
query82 953 87 83 83
query83 236 120 119 119
query84 275 78 77 77
query85 1339 350 343 343
query86 377 261 311 261
query87 3321 3097 3133 3097
query88 2724 2341 2351 2341
query89 389 326 318 318
query90 1843 160 159 159
query91 167 125 123 123
query92 53 49 49 49
query93 1199 524 503 503
query94 1277 175 178 175
query95 438 329 334 329
query96 577 268 265 265
query97 4261 4140 4106 4106
query98 212 206 193 193
query99 1052 642 652 642
Total cold run time: 264173 ms
Total hot run time: 170099 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966112321
<details>
<summary>ClickBench: <b>Total hot run time: 31.89 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.24 0.05 0.06
query4 1.67 0.10 0.10
query5 0.52 0.51 0.51
query6 1.19 0.68 0.67
query7 0.02 0.01 0.02
query8 0.04 0.03 0.02
query9 0.56 0.51 0.50
query10 0.54 0.61 0.58
query11 0.12 0.09 0.10
query12 0.13 0.10 0.10
query13 0.62 0.62 0.60
query14 0.78 0.80 0.81
query15 0.84 0.82 0.81
query16 0.39 0.38 0.38
query17 0.95 0.96 0.95
query18 0.26 0.26 0.24
query19 1.76 1.69 1.75
query20 0.02 0.01 0.01
query21 15.44 0.64 0.65
query22 3.23 4.56 2.14
query23 17.68 1.13 0.96
query24 2.18 0.40 0.38
query25 0.45 0.19 0.08
query26 0.17 0.17 0.16
query27 0.04 0.05 0.03
query28 12.05 0.83 0.82
query29 12.58 3.31 3.18
query30 0.51 0.53 0.57
query31 2.78 0.36 0.36
query32 3.35 0.48 0.48
query33 3.18 3.19 3.23
query34 15.36 4.76 4.75
query35 4.75 4.74 4.71
query36 1.14 1.09 1.07
query37 0.06 0.04 0.04
query38 0.03 0.02 0.03
query39 0.02 0.01 0.02
query40 0.17 0.13 0.13
query41 0.07 0.02 0.02
query42 0.02 0.02 0.01
query43 0.02 0.03 0.02
Total cold run time: 106.03 s
Total hot run time: 31.89 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966498303
<details>
<summary>ClickBench: <b>Total hot run time: 32.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e873221470293890959c7989f764d15b44005114, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.09 0.10
query5 0.52 0.51 0.52
query6 1.29 0.67 0.68
query7 0.02 0.02 0.01
query8 0.03 0.02 0.02
query9 0.55 0.54 0.52
query10 0.57 0.56 0.57
query11 0.13 0.10 0.10
query12 0.12 0.11 0.10
query13 0.62 0.61 0.61
query14 0.78 0.80 0.81
query15 0.83 0.84 0.82
query16 0.40 0.37 0.36
query17 1.00 0.96 0.97
query18 0.26 0.25 0.26
query19 1.79 1.72 1.73
query20 0.01 0.01 0.02
query21 15.40 0.65 0.63
query22 3.48 3.63 2.60
query23 17.30 1.07 0.91
query24 1.97 0.59 0.56
query25 0.47 0.14 0.06
query26 0.17 0.14 0.17
query27 0.05 0.04 0.04
query28 11.88 0.83 0.82
query29 12.59 3.31 3.30
query30 0.56 0.53 0.46
query31 2.78 0.35 0.36
query32 3.35 0.48 0.49
query33 3.17 3.23 3.19
query34 15.34 4.77 4.78
query35 4.81 4.78 4.77
query36 1.14 1.07 1.10
query37 0.07 0.06 0.05
query38 0.05 0.02 0.02
query39 0.02 0.02 0.01
query40 0.17 0.13 0.13
query41 0.08 0.01 0.02
query42 0.03 0.01 0.02
query43 0.02 0.03 0.02
Total cold run time: 105.82 s
Total hot run time: 32.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966033213
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966033498
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966099911
<details>
<summary>TPC-DS: <b>Total hot run time: 169101 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
query1 926 339 334 334
query2 6545 1898 1780 1780
query3 6706 208 210 208
query4 23494 20614 20437 20437
query5 4324 359 355 355
query6 258 167 181 167
query7 4613 292 291 291
query8 258 192 202 192
query9 8516 2309 2306 2306
query10 412 207 222 207
query11 14671 14241 14114 14114
query12 140 84 81 81
query13 1631 429 454 429
query14 8290 6639 6339 6339
query15 221 177 184 177
query16 7091 261 255 255
query17 1033 514 494 494
query18 1898 268 260 260
query19 190 147 150 147
query20 88 81 84 81
query21 195 127 124 124
query22 4808 4595 4529 4529
query23 30552 29881 29914 29881
query24 11781 3088 3121 3088
query25 648 376 356 356
query26 1835 149 152 149
query27 3067 316 324 316
query28 7045 1865 1850 1850
query29 1198 554 550 550
query30 284 129 140 129
query31 863 680 703 680
query32 95 58 62 58
query33 715 220 222 220
query34 1061 474 499 474
query35 840 749 745 745
query36 972 868 877 868
query37 163 65 59 59
query38 3120 3010 2986 2986
query39 1301 1258 1247 1247
query40 280 104 106 104
query41 37 37 36 36
query42 107 99 97 97
query43 479 437 432 432
query44 1060 704 725 704
query45 189 178 176 176
query46 1057 794 792 792
query47 1533 1428 1389 1389
query48 407 349 353 349
query49 1159 289 288 288
query50 769 377 378 377
query51 4379 4249 4305 4249
query52 114 87 91 87
query53 332 264 265 264
query54 283 229 232 229
query55 84 79 85 79
query56 220 202 195 195
query57 967 890 879 879
query58 213 192 189 189
query59 2307 1973 2304 1973
query60 239 226 221 221
query61 90 82 83 82
query62 590 353 331 331
query63 279 256 275 256
query64 6545 3061 3060 3060
query65 3226 3212 3209 3209
query66 1431 318 309 309
query67 14573 13959 14148 13959
query68 4951 558 567 558
query69 529 371 379 371
query70 1262 1191 1217 1191
query71 352 256 250 250
query72 6213 2795 2622 2622
query73 723 330 321 321
query74 6230 5991 5970 5970
query75 3087 2446 2409 2409
query76 2822 1062 1162 1062
query77 330 230 235 230
query78 8927 8702 8497 8497
query79 969 539 522 522
query80 509 361 345 345
query81 454 201 194 194
query82 158 83 83 83
query83 203 120 118 118
query84 273 81 74 74
query85 1032 343 337 337
query86 295 299 310 299
query87 3221 3126 3080 3080
query88 2736 2380 2344 2344
query89 383 333 334 333
query90 1821 164 170 164
query91 152 122 124 122
query92 53 51 49 49
query93 1022 526 506 506
query94 1058 179 180 179
query95 440 341 342 341
query96 562 267 265 265
query97 4256 4123 4166 4123
query98 222 200 191 191
query99 1012 673 700 673
Total cold run time: 261054 ms
Total hot run time: 169101 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31448:
URL: https://github.com/apache/doris/pull/31448#discussion_r1504138819
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
warning: function 'do_report_profile' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/task_execution_context.cpp:54:** 233 lines including whitespace and comments (threshold 80)
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
</details>
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,290 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
warning: function 'do_report_profile' has cognitive complexity of 53 (threshold 50) [readability-function-cognitive-complexity]
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/task_execution_context.cpp:57:** +1
```cpp
DCHECK(req->status.ok() || req->done); // if !status.ok() => done
^
```
**be/src/runtime/task_execution_context.cpp:62:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!coord_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:84:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (req->runtime_state->query_type() == TQueryType::LOAD && !req->done && req->status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:88:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:89:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->runtime_state->query_type() == TQueryType::LOAD) {
^
```
**be/src/runtime/task_execution_context.cpp:93:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->is_pipeline_x) {
^
```
**be/src/runtime/task_execution_context.cpp:97:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (enable_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:108:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:112:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (enable_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:122:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:123:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (req->profile != nullptr) {
^
```
**be/src/runtime/task_execution_context.cpp:125:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (req->load_channel_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:130:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:135:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->output_files().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:140:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:146:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!params.delta_urls.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:158:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->runtime_state->num_rows_load_total() > 0 ||
^
```
**be/src/runtime/task_execution_context.cpp:158:** +1
```cpp
if (req->runtime_state->num_rows_load_total() > 0 ||
^
```
**be/src/runtime/task_execution_context.cpp:165:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:179:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->get_error_log_file_path().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:182:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:190:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->export_output_files().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:193:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:203:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->tablet_commit_infos().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:209:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:219:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->error_tablet_infos().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:225:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:241:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (ExecEnv::GetInstance()->master_info()->__isset.backend_id) {
^
```
**be/src/runtime/task_execution_context.cpp:250:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!exec_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:259:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
} catch (TTransportException& e) {
^
```
**be/src/runtime/task_execution_context.cpp:265:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!rpc_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:275:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
} catch (TException& e) {
^
```
**be/src/runtime/task_execution_context.cpp:280:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!rpc_status.ok()) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966410175
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966466805
<details>
<summary>TPC-H: <b>Total hot run time: 38001 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e873221470293890959c7989f764d15b44005114, data reload: false
------ Round 1 ----------------------------------
q1 17712 4071 4057 4057
q2 2056 139 140 139
q3 10571 949 941 941
q4 4659 937 943 937
q5 7628 2920 3015 2920
q6 181 122 121 121
q7 1230 766 752 752
q8 9249 2004 2024 2004
q9 7175 6294 6281 6281
q10 8192 2495 2485 2485
q11 418 198 202 198
q12 760 318 313 313
q13 17974 3215 3212 3212
q14 282 256 254 254
q15 534 498 489 489
q16 476 410 394 394
q17 950 899 887 887
q18 6627 5921 6052 5921
q19 1562 1509 1520 1509
q20 542 272 270 270
q21 7015 3656 3630 3630
q22 803 312 287 287
Total cold run time: 106596 ms
Total hot run time: 38001 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4098 4141 4063 4063
q2 305 215 216 215
q3 2986 2917 2967 2917
q4 1819 1831 1841 1831
q5 5109 5117 5149 5117
q6 198 114 117 114
q7 2149 1689 1770 1689
q8 3152 3186 3213 3186
q9 8298 8261 8247 8247
q10 6131 3650 3654 3650
q11 510 428 433 428
q12 687 564 532 532
q13 9559 3066 3048 3048
q14 277 260 268 260
q15 529 488 500 488
q16 516 458 489 458
q17 1699 1684 1698 1684
q18 7989 7439 7576 7439
q19 1667 1672 1632 1632
q20 2100 1915 1893 1893
q21 4816 4569 4530 4530
q22 512 416 414 414
Total cold run time: 65106 ms
Total hot run time: 53835 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31448:
URL: https://github.com/apache/doris/pull/31448#discussion_r1503649789
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
warning: method 'do_report_profile' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void do_report_profile(std::shared_ptr<doris::ReportStatusRequest> request);
```
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
Review Comment:
warning: method 'trigger_profile_report' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/task_execution_context.h:51:
```diff
- Status trigger_profile_report(std::shared_ptr<doris::ReportStatusRequest> request);
+ static Status trigger_profile_report(std::shared_ptr<doris::ReportStatusRequest> request);
```
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
warning: function 'do_report_profile' has cognitive complexity of 53 (threshold 50) [readability-function-cognitive-complexity]
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/task_execution_context.cpp:58:** +1
```cpp
DCHECK(req->status.ok() || req->done); // if !status.ok() => done
^
```
**be/src/runtime/task_execution_context.cpp:63:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!coord_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:85:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (req->runtime_state->query_type() == TQueryType::LOAD && !req->done && req->status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:89:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:90:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->runtime_state->query_type() == TQueryType::LOAD) {
^
```
**be/src/runtime/task_execution_context.cpp:94:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->is_pipeline_x) {
^
```
**be/src/runtime/task_execution_context.cpp:98:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (enable_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:109:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:113:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (enable_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:123:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:124:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (req->profile != nullptr) {
^
```
**be/src/runtime/task_execution_context.cpp:126:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
if (req->load_channel_profile) {
^
```
**be/src/runtime/task_execution_context.cpp:131:** +1, nesting level increased to 3
```cpp
} else {
^
```
**be/src/runtime/task_execution_context.cpp:136:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->output_files().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:141:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:147:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
if (!params.delta_urls.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:159:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (req->runtime_state->num_rows_load_total() > 0 ||
^
```
**be/src/runtime/task_execution_context.cpp:159:** +1
```cpp
if (req->runtime_state->num_rows_load_total() > 0 ||
^
```
**be/src/runtime/task_execution_context.cpp:166:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:180:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->get_error_log_file_path().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:183:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:191:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->export_output_files().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:194:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:204:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->tablet_commit_infos().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:210:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:220:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!req->runtime_state->error_tablet_infos().empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:226:** +1, nesting level increased to 2
```cpp
} else if (!req->runtime_states.empty()) {
^
```
**be/src/runtime/task_execution_context.cpp:242:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (ExecEnv::GetInstance()->master_info()->__isset.backend_id) {
^
```
**be/src/runtime/task_execution_context.cpp:251:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!exec_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:260:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
} catch (TTransportException& e) {
^
```
**be/src/runtime/task_execution_context.cpp:266:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!rpc_status.ok()) {
^
```
**be/src/runtime/task_execution_context.cpp:276:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
} catch (TException& e) {
^
```
**be/src/runtime/task_execution_context.cpp:281:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!rpc_status.ok()) {
^
```
</details>
##########
be/src/runtime/task_execution_context.cpp:
##########
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/task_execution_context.h"
+
+#include <thrift/protocol/TDebugProtocol.h>
+#include <thrift/transport/TTransportException.h>
+
+#include <memory>
+
+#include "runtime/client_cache.h"
+#include "runtime/exec_env.h"
+#include "runtime/fragment_mgr.h"
+#include "service/backend_options.h"
+
+namespace doris {
+std::string to_http_path(const std::string& file_name) {
+ std::stringstream url;
+ url << "http://" << BackendOptions::get_localhost() << ":" << config::webserver_port
+ << "/api/_download_load?"
+ << "token=" << ExecEnv::GetInstance()->token() << "&file=" << file_name;
+ return url.str();
+};
+
+Status TaskExecutionContext::trigger_profile_report(
+ std::shared_ptr<doris::ReportStatusRequest> request) {
+ // Create a shared pointer to capture and extend the lifetime of TaskExecutionContext
+ auto shared_this = shared_from_this();
+
+ auto report_profile_task = [shared_this, request]() {
+ shared_this->do_report_profile(request);
+
+ if (!request->done) {
+ shared_this->refresh_next_report_time();
+ }
+ };
+
+ return ExecEnv::GetInstance()->fragment_mgr()->submit_report_profile_task(report_profile_task);
+};
+
+void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
Review Comment:
warning: function 'do_report_profile' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/task_execution_context.cpp:54:** 234 lines including whitespace and comments (threshold 80)
```cpp
void TaskExecutionContext::do_report_profile(std::shared_ptr<doris::ReportStatusRequest> req) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](profile callback) refactor profile callback on BE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31448:
URL: https://github.com/apache/doris/pull/31448#issuecomment-1966081050
<details>
<summary>TPC-H: <b>Total hot run time: 38129 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1a1fe67d9c8d9df01671b4ff2f904e1fa820a78, data reload: false
------ Round 1 ----------------------------------
q1 17645 4153 4046 4046
q2 2045 137 135 135
q3 11255 982 961 961
q4 5401 956 947 947
q5 7828 2972 3090 2972
q6 181 121 124 121
q7 1282 775 783 775
q8 10287 2067 2093 2067
q9 7852 6321 6300 6300
q10 8195 2490 2511 2490
q11 410 200 206 200
q12 727 306 315 306
q13 17956 3213 3243 3213
q14 294 244 253 244
q15 536 500 492 492
q16 467 428 403 403
q17 957 902 886 886
q18 6596 6074 5910 5910
q19 1576 1516 1518 1516
q20 549 271 277 271
q21 6961 3631 3590 3590
q22 807 284 290 284
Total cold run time: 109807 ms
Total hot run time: 38129 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4027 4023 4033 4023
q2 311 223 212 212
q3 2979 2944 2922 2922
q4 1832 1842 1824 1824
q5 5176 5158 5141 5141
q6 202 115 116 115
q7 2125 1727 1723 1723
q8 3144 3205 3226 3205
q9 8256 8295 8264 8264
q10 6197 3624 3646 3624
q11 498 419 438 419
q12 693 551 540 540
q13 3852 3046 3045 3045
q14 292 255 244 244
q15 554 491 494 491
q16 504 471 466 466
q17 1698 1674 1678 1674
q18 7726 7714 7395 7395
q19 1712 1663 1641 1641
q20 2099 1897 1916 1897
q21 4795 4688 4590 4590
q22 532 432 420 420
Total cold run time: 59204 ms
Total hot run time: 53875 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org