You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/01/26 09:15:48 UTC
[PR] [improve](stream-load) record time on receiving HTTP request [doris]
HHoflittlefish777 opened a new pull request, #30432:
URL: https://github.com/apache/doris/pull/30432
## Proposed changes
We meet occasional slow stream-load, ReadDataTimeMs is very fast but WriteDataTimeMs is too long, according to some logs, it can be inferred that data from receive to vtablet_writer is slow, but can not judge the root cause is the scanner slow or is it slow to receive data. Finally, finding the network is too slow, but it costs lots of time. Therefore, recording the time on receiving HTTP request is helpful to users to know why the load is slow.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1467562899
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -148,6 +148,13 @@ Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadConte
static_cast<void>(this->commit_txn(ctx.get()));
}
}
+
+ LOG(INFO) << "finished to execute job. label=" << ctx->label << ", txn_id=" << ctx->txn_id
Review Comment:
> leave a keyword such as streamload in the log.
I will update it.
> and consider adding a bvar latency recorder for streamload
Perhaps bvar needs to be on-site, but without it, we may not know what happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1467562899
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -148,6 +148,13 @@ Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadConte
static_cast<void>(this->commit_txn(ctx.get()));
}
}
+
+ LOG(INFO) << "finished to execute job. label=" << ctx->label << ", txn_id=" << ctx->txn_id
Review Comment:
> leave a keyword such as streamload in the log.
I will update it.
> and consider adding a bvar latency recorder for streamload
Perhaps bvar needs to be on-site, but without it, we may not know what happened?
Maybe both log and bvar is all need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) add observability on receiving HTTP request [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30432:
URL: https://github.com/apache/doris/pull/30432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916705756
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916660371
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) add observability on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1920613276
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913107415
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911770048
<details>
<summary>TPC-H: <b>Total hot run time: 39270 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false
------ Round 1 ----------------------------------
q1 17661 5803 5292 5292
q2 2054 146 132 132
q3 10600 978 947 947
q4 4642 770 826 770
q5 7689 2928 3009 2928
q6 186 121 124 121
q7 1089 739 716 716
q8 9262 2025 2028 2025
q9 7263 6418 6392 6392
q10 8103 2460 2428 2428
q11 421 223 201 201
q12 744 298 287 287
q13 18006 3336 3367 3336
q14 295 264 253 253
q15 549 515 497 497
q16 471 408 430 408
q17 938 557 557 557
q18 7564 6978 6864 6864
q19 1552 1297 1464 1297
q20 613 349 357 349
q21 6980 3162 3261 3162
q22 808 314 308 308
Total cold run time: 107490 ms
Total hot run time: 39270 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5352 5358 5187 5187
q2 324 223 208 208
q3 3324 3207 3195 3195
q4 2115 2019 2057 2019
q5 5868 5882 5882 5882
q6 199 119 120 119
q7 2293 1902 1850 1850
q8 3198 3382 3391 3382
q9 8933 8899 8726 8726
q10 5290 3929 3824 3824
q11 563 453 439 439
q12 814 653 611 611
q13 16897 3160 3135 3135
q14 294 271 265 265
q15 553 498 496 496
q16 531 460 472 460
q17 1854 1818 1795 1795
q18 9532 19508 9636 9636
q19 24118 1580 1528 1528
q20 4621 1983 1911 1911
q21 14490 5416 5299 5299
q22 969 521 553 521
Total cold run time: 112132 ms
Total hot run time: 60488 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911995856
<details>
<summary>ClickBench: <b>Total hot run time: 30.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.03
query3 0.23 0.05 0.04
query4 1.68 0.06 0.06
query5 0.54 0.53 0.52
query6 1.24 0.65 0.62
query7 0.02 0.01 0.01
query8 0.03 0.03 0.03
query9 0.54 0.50 0.48
query10 0.55 0.54 0.55
query11 0.12 0.09 0.08
query12 0.11 0.09 0.09
query13 0.61 0.60 0.61
query14 0.77 0.82 0.78
query15 0.79 0.77 0.77
query16 0.37 0.39 0.38
query17 1.03 0.97 1.03
query18 0.25 0.23 0.23
query19 1.83 1.81 1.76
query20 0.01 0.01 0.01
query21 15.42 0.58 0.58
query22 2.34 2.17 2.14
query23 17.19 0.76 0.78
query24 2.91 0.70 1.37
query25 0.40 0.26 0.10
query26 0.47 0.13 0.14
query27 0.06 0.04 0.06
query28 11.35 0.76 0.75
query29 12.52 3.08 3.16
query30 0.54 0.50 0.52
query31 2.78 0.34 0.34
query32 3.37 0.49 0.49
query33 3.25 3.16 3.21
query34 15.84 4.33 4.31
query35 4.38 4.32 4.23
query36 1.10 1.07 1.08
query37 0.07 0.05 0.05
query38 0.03 0.03 0.02
query39 0.02 0.02 0.01
query40 0.15 0.15 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.11 s
Total hot run time: 30.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913509325
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912959693
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit cbcb6998621b798b06ec737f0a296f186841b677 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.2 seconds inserted 10000000 Rows, about 704K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913133525
<details>
<summary>TPC-H: <b>Total hot run time: 37492 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41c40111a1ccd9485824fd01c0a23cd5cb8075c2, data reload: false
------ Round 1 ----------------------------------
q1 17730 5660 4826 4826
q2 2044 141 128 128
q3 10700 929 953 929
q4 4694 738 756 738
q5 7687 2794 2825 2794
q6 184 123 119 119
q7 1137 726 722 722
q8 9303 1987 2029 1987
q9 7239 6360 6328 6328
q10 8074 2421 2392 2392
q11 409 195 195 195
q12 730 270 276 270
q13 18011 3300 3296 3296
q14 274 249 247 247
q15 523 513 486 486
q16 471 402 401 401
q17 940 546 507 507
q18 7181 6269 6194 6194
q19 1576 1392 1339 1339
q20 616 349 304 304
q21 6625 3010 3091 3010
q22 812 304 280 280
Total cold run time: 106960 ms
Total hot run time: 37492 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5130 4862 4813 4813
q2 338 246 246 246
q3 2986 2730 2679 2679
q4 1796 1565 1474 1474
q5 5163 5194 5269 5194
q6 202 115 116 115
q7 2144 1784 1881 1784
q8 3120 3166 3242 3166
q9 8669 8603 12431 8603
q10 5329 3373 3458 3373
q11 547 451 463 451
q12 768 563 589 563
q13 16946 3098 3055 3055
q14 277 247 268 247
q15 529 483 489 483
q16 516 464 469 464
q17 2046 1602 1692 1602
q18 9094 9436 8526 8526
q19 23915 1564 1497 1497
q20 4602 1929 1889 1889
q21 14786 4809 4959 4809
q22 1023 455 479 455
Total cold run time: 109926 ms
Total hot run time: 55488 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912948076
<details>
<summary>TPC-H: <b>Total hot run time: 37404 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false
------ Round 1 ----------------------------------
q1 17659 5067 4871 4871
q2 2041 148 138 138
q3 10686 933 923 923
q4 4711 770 730 730
q5 7714 2876 2740 2740
q6 187 118 119 118
q7 1121 727 709 709
q8 9362 2036 2006 2006
q9 7160 6295 6317 6295
q10 8087 2420 2421 2420
q11 414 200 205 200
q12 785 287 282 282
q13 18018 3242 3275 3242
q14 273 246 243 243
q15 536 518 489 489
q16 456 408 403 403
q17 931 569 516 516
q18 6786 6285 6025 6025
q19 1565 1458 1377 1377
q20 626 339 365 339
q21 7130 3039 3087 3039
q22 811 299 300 299
Total cold run time: 107059 ms
Total hot run time: 37404 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5191 5008 4985 4985
q2 332 242 236 236
q3 3025 2753 2709 2709
q4 1781 1504 1514 1504
q5 5215 5230 5395 5230
q6 198 114 115 114
q7 2201 1837 1804 1804
q8 3097 3204 3204 3204
q9 8703 8689 11929 8689
q10 5599 3411 3478 3411
q11 556 462 477 462
q12 712 610 561 561
q13 16958 3052 3102 3052
q14 283 253 262 253
q15 530 488 486 486
q16 524 470 453 453
q17 1863 1765 1681 1681
q18 8951 10923 8629 8629
q19 23991 1572 1559 1559
q20 4591 1922 1904 1904
q21 15278 4816 4730 4730
q22 1124 459 472 459
Total cold run time: 110703 ms
Total hot run time: 56115 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913099988
TeamCity be ut coverage result:
Function Coverage: 36.24% (8611/23763)
Line Coverage: 28.31% (70479/248932)
Region Coverage: 27.28% (36359/133258)
Branch Coverage: 24.09% (18641/77382)
Coverage Report: http://coverage.selectdb-in.cc/coverage/caabe770ff415d2581d21754c8f962300481107c_caabe770ff415d2581d21754c8f962300481107c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911999380
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913485611
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1467557045
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -148,6 +148,13 @@ Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadConte
static_cast<void>(this->commit_txn(ctx.get()));
}
}
+
+ LOG(INFO) << "finished to execute job. label=" << ctx->label << ", txn_id=" << ctx->txn_id
Review Comment:
leave a keyword such as streamload in the log.
and consider adding a bvar latency recorder for streamload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911946925
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911988901
<details>
<summary>TPC-DS: <b>Total hot run time: 175860 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false
query1 931 333 322 322
query2 6568 2010 1806 1806
query3 6700 196 195 195
query4 30795 22343 22135 22135
query5 4452 430 367 367
query6 248 151 145 145
query7 4602 260 260 260
query8 222 169 181 169
query9 8440 2517 2509 2509
query10 407 217 200 200
query11 16817 15421 15449 15421
query12 120 67 68 67
query13 1681 368 384 368
query14 9731 6656 6672 6656
query15 209 181 187 181
query16 5695 253 241 241
query17 956 481 460 460
query18 1783 244 250 244
query19 181 138 124 124
query20 70 64 67 64
query21 195 132 127 127
query22 5193 5031 5049 5031
query23 31319 30813 30556 30556
query24 11627 2759 2762 2759
query25 584 308 308 308
query26 1728 135 136 135
query27 3199 282 280 280
query28 7640 1838 1821 1821
query29 1680 622 610 610
query30 273 136 136 136
query31 974 727 749 727
query32 70 53 45 45
query33 698 211 204 204
query34 1138 466 465 465
query35 871 776 748 748
query36 1370 1224 1243 1224
query37 92 59 63 59
query38 3330 3246 3225 3225
query39 1300 1247 1235 1235
query40 201 85 86 85
query41 38 39 38 38
query42 88 77 82 77
query43 546 484 505 484
query44 1095 686 685 685
query45 188 182 174 174
query46 1064 658 664 658
query47 1677 1575 1552 1552
query48 393 318 304 304
query49 1138 279 275 275
query50 666 313 300 300
query51 5236 5233 5189 5189
query52 90 78 78 78
query53 321 251 255 251
query54 235 186 175 175
query55 75 79 82 79
query56 167 165 147 147
query57 1013 893 926 893
query58 181 156 149 149
query59 2988 2801 2770 2770
query60 201 174 173 173
query61 87 81 82 81
query62 614 345 365 345
query63 287 260 271 260
query64 4971 1757 1735 1735
query65 3294 3260 3251 3251
query66 1326 319 307 307
query67 15652 15112 15068 15068
query68 16459 501 483 483
query69 600 279 286 279
query70 2291 1500 1503 1500
query71 508 215 201 201
query72 5018 2801 2812 2801
query73 4483 318 303 303
query74 7003 6395 6401 6395
query75 5020 2336 2292 2292
query76 6270 1020 1018 1018
query77 760 247 229 229
query78 9772 8821 8713 8713
query79 2432 489 498 489
query80 638 315 310 310
query81 483 211 204 204
query82 201 83 77 77
query83 170 113 109 109
query84 278 65 67 65
query85 1106 341 322 322
query86 380 397 378 378
query87 3500 3321 3306 3306
query88 2799 2198 2188 2188
query89 421 352 338 338
query90 2012 183 187 183
query91 151 114 126 114
query92 51 43 47 43
query93 1287 437 408 408
query94 1360 154 160 154
query95 501 447 436 436
query96 605 322 318 318
query97 4261 4123 4163 4123
query98 193 187 182 182
query99 1063 698 687 687
Total cold run time: 299797 ms
Total hot run time: 175860 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1467562899
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -148,6 +148,13 @@ Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadConte
static_cast<void>(this->commit_txn(ctx.get()));
}
}
+
+ LOG(INFO) << "finished to execute job. label=" << ctx->label << ", txn_id=" << ctx->txn_id
Review Comment:
> leave a keyword such as streamload in the log.
I will update it.
> and consider adding a bvar latency recorder for streamload
Perhaps bvar needs to be on-site, but without it, we may not know what happened?
Maybe both log and bvar are needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913091274
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916642799
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913487930
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916691197
<details>
<summary>TPC-DS: <b>Total hot run time: 172986 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false
query1 928 338 319 319
query2 6566 2094 1927 1927
query3 6697 208 208 208
query4 28155 22107 22025 22025
query5 4452 355 359 355
query6 245 162 167 162
query7 4603 270 261 261
query8 245 186 195 186
query9 9128 2241 2231 2231
query10 404 199 203 199
query11 16701 15294 15293 15293
query12 122 67 60 60
query13 1668 370 370 370
query14 9351 6550 6427 6427
query15 229 178 186 178
query16 6525 254 253 253
query17 1903 483 462 462
query18 1935 248 248 248
query19 213 131 135 131
query20 72 71 69 69
query21 203 131 124 124
query22 5095 4949 4917 4917
query23 31279 30219 30412 30219
query24 12354 2719 2778 2719
query25 583 310 303 303
query26 1991 138 133 133
query27 2900 288 284 284
query28 7627 1837 1802 1802
query29 2169 626 604 604
query30 285 137 138 137
query31 890 707 712 707
query32 90 55 51 51
query33 707 205 207 205
query34 1067 449 464 449
query35 855 774 758 758
query36 1326 1167 1245 1167
query37 176 56 56 56
query38 3252 3161 3120 3120
query39 1306 1251 1252 1251
query40 325 79 81 79
query41 37 36 33 33
query42 86 84 83 83
query43 546 507 515 507
query44 1100 682 673 673
query45 188 179 173 173
query46 1039 631 625 625
query47 1547 1546 1498 1498
query48 386 304 303 303
query49 1269 281 297 281
query50 678 313 311 311
query51 5280 5122 5163 5122
query52 90 79 82 79
query53 332 251 269 251
query54 253 188 187 187
query55 81 76 74 74
query56 178 168 156 156
query57 1011 898 906 898
query58 178 152 153 152
query59 2356 2399 2402 2399
query60 196 179 178 178
query61 86 83 82 82
query62 629 362 343 343
query63 277 263 258 258
query64 5567 1760 1726 1726
query65 3282 3216 3189 3189
query66 1491 322 310 310
query67 14907 14347 14418 14347
query68 3951 497 492 492
query69 414 305 314 305
query70 1559 1565 1492 1492
query71 303 221 230 221
query72 3697 2836 2854 2836
query73 695 321 317 317
query74 6548 6152 6146 6146
query75 2819 2291 2290 2290
query76 2899 933 926 926
query77 322 232 228 228
query78 9059 8840 8593 8593
query79 2131 483 481 481
query80 1335 333 324 324
query81 532 199 195 195
query82 674 77 76 76
query83 235 121 111 111
query84 234 72 72 72
query85 1988 326 334 326
query86 494 386 358 358
query87 3388 3293 3243 3243
query88 3806 2265 2231 2231
query89 432 350 343 343
query90 1936 186 189 186
query91 152 117 116 116
query92 52 45 41 41
query93 1945 437 434 434
query94 1364 157 156 156
query95 490 466 452 452
query96 623 330 321 321
query97 4261 4127 4143 4127
query98 203 198 186 186
query99 948 674 677 674
Total cold run time: 277181 ms
Total hot run time: 172986 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911785972
<details>
<summary>TPC-DS: <b>Total hot run time: 176242 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false
query1 932 335 323 323
query2 6576 1937 1924 1924
query3 6698 201 199 199
query4 31458 22402 22183 22183
query5 4453 439 442 439
query6 253 155 153 153
query7 4606 269 259 259
query8 222 176 177 176
query9 8392 2538 2520 2520
query10 415 242 232 232
query11 16994 15689 15559 15559
query12 121 69 65 65
query13 1682 382 381 381
query14 9794 6899 6750 6750
query15 211 177 190 177
query16 5699 253 250 250
query17 958 475 478 475
query18 1802 260 249 249
query19 176 134 131 131
query20 73 71 67 67
query21 188 133 130 130
query22 4912 4473 4743 4473
query23 31478 31083 30803 30803
query24 11592 2858 2773 2773
query25 589 311 311 311
query26 1763 150 138 138
query27 3172 287 287 287
query28 7510 1821 1805 1805
query29 1805 639 609 609
query30 273 134 137 134
query31 952 724 752 724
query32 66 47 50 47
query33 691 210 213 210
query34 1134 456 465 456
query35 883 764 791 764
query36 1273 1223 1223 1223
query37 93 59 61 59
query38 3356 3269 3259 3259
query39 1305 1266 1275 1266
query40 343 84 85 84
query41 36 35 34 34
query42 90 83 78 78
query43 513 465 445 445
query44 1053 690 693 690
query45 196 174 176 174
query46 1049 665 648 648
query47 1588 1525 1549 1525
query48 407 323 310 310
query49 1247 293 287 287
query50 696 312 310 310
query51 5274 5252 5213 5213
query52 86 75 74 74
query53 329 260 261 260
query54 252 185 184 184
query55 81 73 79 73
query56 178 168 162 162
query57 973 922 951 922
query58 176 147 151 147
query59 2978 2788 2470 2470
query60 208 179 180 179
query61 83 91 85 85
query62 636 360 350 350
query63 292 260 252 252
query64 6011 1805 1777 1777
query65 3332 3260 3251 3251
query66 1298 317 315 315
query67 15619 15282 15259 15259
query68 10916 535 546 535
query69 597 293 299 293
query70 1683 1459 1542 1459
query71 536 212 227 212
query72 4846 2848 2866 2848
query73 2127 325 312 312
query74 6982 6385 6316 6316
query75 4690 2352 2323 2323
query76 6737 1032 1036 1032
query77 662 231 236 231
query78 9017 9017 8549 8549
query79 1009 509 488 488
query80 591 315 315 315
query81 444 196 197 196
query82 203 80 74 74
query83 136 118 117 117
query84 273 69 67 67
query85 1122 325 333 325
query86 381 381 403 381
query87 3459 3370 3316 3316
query88 3298 2224 2231 2224
query89 437 357 355 355
query90 2019 187 184 184
query91 143 116 128 116
query92 52 43 43 43
query93 3212 462 415 415
query94 1263 158 162 158
query95 496 451 460 451
query96 638 318 316 316
query97 4268 4151 4179 4151
query98 203 195 191 191
query99 1029 708 718 708
Total cold run time: 293043 ms
Total hot run time: 176242 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912930317
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1468293006
##########
be/src/http/action/stream_load.cpp:
##########
@@ -351,6 +354,11 @@ void StreamLoadAction::on_chunk_data(HttpRequest* req) {
ctx->receive_bytes += remove_bytes;
}
ctx->read_data_cost_nanos += (MonotonicNanos() - start_read_data_time);
+ int64_t last_receive_and_read_data_cost_nanos = ctx->receive_and_read_data_cost_nanos;
+ ctx->receive_and_read_data_cost_nanos =
+ MonotonicNanos() - last_receive_and_read_data_cost_nanos;
+ g_stream_load_receive_data_latency
+ << ctx->receive_and_read_data_cost_nanos - ctx->read_data_cost_nanos;
Review Comment:
latency shoud be in microseconds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913138725
<details>
<summary>ClickBench: <b>Total hot run time: 30.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41c40111a1ccd9485824fd01c0a23cd5cb8075c2, data reload: false
query1 0.05 0.03 0.03
query2 0.06 0.02 0.02
query3 0.22 0.06 0.07
query4 1.68 0.10 0.10
query5 0.52 0.52 0.51
query6 1.20 0.65 0.62
query7 0.02 0.02 0.01
query8 0.04 0.03 0.02
query9 0.54 0.48 0.50
query10 0.54 0.55 0.54
query11 0.12 0.09 0.09
query12 0.11 0.09 0.10
query13 0.61 0.60 0.61
query14 0.79 0.81 0.78
query15 0.78 0.76 0.78
query16 0.39 0.38 0.40
query17 1.04 1.04 1.04
query18 0.22 0.26 0.25
query19 1.90 1.73 1.72
query20 0.01 0.01 0.01
query21 15.40 0.55 0.56
query22 2.31 2.44 2.03
query23 17.27 0.91 0.72
query24 2.57 1.20 0.44
query25 0.34 0.09 0.10
query26 0.58 0.13 0.12
query27 0.04 0.06 0.04
query28 11.79 0.84 0.85
query29 12.50 3.30 3.17
query30 0.68 0.55 0.58
query31 2.80 0.35 0.35
query32 3.36 0.49 0.48
query33 3.20 3.20 3.25
query34 15.84 4.22 4.37
query35 4.36 4.35 4.37
query36 1.10 1.05 1.05
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.18 0.13 0.14
query41 0.07 0.01 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.03
Total cold run time: 105.41 s
Total hot run time: 30.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911936870
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913506911
<details>
<summary>TPC-DS: <b>Total hot run time: 172596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false
query1 950 332 326 326
query2 6567 1858 1937 1858
query3 6696 198 192 192
query4 31159 22136 22098 22098
query5 4471 429 363 363
query6 253 152 149 149
query7 4614 259 251 251
query8 235 175 167 167
query9 8553 2272 2253 2253
query10 425 198 209 198
query11 17882 15513 15246 15246
query12 124 69 65 65
query13 1683 377 361 361
query14 9130 6402 6416 6402
query15 205 179 179 179
query16 5694 257 247 247
query17 953 474 459 459
query18 1804 260 253 253
query19 172 131 126 126
query20 74 72 70 70
query21 200 129 137 129
query22 4918 4707 4640 4640
query23 31205 30361 30313 30313
query24 12435 2728 2758 2728
query25 593 312 314 312
query26 1940 141 132 132
query27 3199 276 279 276
query28 7771 1831 1819 1819
query29 2010 624 606 606
query30 281 134 135 134
query31 901 722 748 722
query32 78 52 56 52
query33 699 205 203 203
query34 1177 456 464 456
query35 863 750 758 750
query36 1381 1213 1183 1183
query37 93 60 62 60
query38 3313 3157 3168 3157
query39 1289 1244 1233 1233
query40 345 86 80 80
query41 38 38 35 35
query42 87 83 81 81
query43 505 457 489 457
query44 1123 679 678 678
query45 192 174 172 172
query46 1046 642 655 642
query47 1585 1524 1492 1492
query48 379 319 304 304
query49 1210 282 275 275
query50 678 296 305 296
query51 5309 5133 5168 5133
query52 83 81 71 71
query53 316 251 261 251
query54 239 179 185 179
query55 79 82 76 76
query56 166 161 162 161
query57 1008 884 890 884
query58 174 153 153 153
query59 2734 2457 2451 2451
query60 201 170 170 170
query61 89 83 82 82
query62 641 336 372 336
query63 275 257 262 257
query64 5998 1775 1757 1757
query65 3294 3224 3225 3224
query66 1348 319 317 317
query67 14368 13997 14223 13997
query68 3907 479 489 479
query69 434 300 312 300
query70 1523 1576 1521 1521
query71 284 210 210 210
query72 3749 2850 2839 2839
query73 683 310 307 307
query74 6653 6229 6256 6229
query75 2771 2336 2291 2291
query76 2616 1026 1054 1026
query77 349 229 236 229
query78 9128 8683 8632 8632
query79 2085 501 487 487
query80 1326 316 310 310
query81 517 194 202 194
query82 679 82 82 82
query83 219 115 113 113
query84 279 72 73 72
query85 2175 341 316 316
query86 513 405 416 405
query87 3425 3333 3248 3248
query88 3926 2142 2152 2142
query89 400 355 339 339
query90 1924 189 182 182
query91 146 114 118 114
query92 55 42 41 41
query93 2193 416 443 416
query94 1342 165 165 165
query95 504 453 447 447
query96 625 306 311 306
query97 4268 4089 4161 4089
query98 209 197 190 190
query99 1003 696 732 696
Total cold run time: 279410 ms
Total hot run time: 172596 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913495939
TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.26% (70530/249532)
Region Coverage: 27.26% (36387/133495)
Branch Coverage: 24.06% (18651/77516)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5b4e7848cc055858ccb3f0249195afba7a880276_5b4e7848cc055858ccb3f0249195afba7a880276/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911725400
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912933710
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913102791
<details>
<summary>TPC-DS: <b>Total hot run time: 173126 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit caabe770ff415d2581d21754c8f962300481107c, data reload: false
query1 929 324 321 321
query2 6574 2037 1916 1916
query3 6705 197 192 192
query4 33474 22060 22078 22060
query5 4453 429 343 343
query6 249 173 158 158
query7 4609 260 255 255
query8 250 182 177 177
query9 8476 2298 2275 2275
query10 433 215 200 200
query11 17900 15486 15373 15373
query12 117 70 66 66
query13 1678 368 367 367
query14 9145 6462 6436 6436
query15 200 174 182 174
query16 5707 250 252 250
query17 955 479 457 457
query18 1810 252 254 252
query19 164 132 127 127
query20 75 69 70 69
query21 196 130 135 130
query22 4951 4771 4763 4763
query23 31263 30626 30391 30391
query24 12475 2789 2816 2789
query25 593 305 313 305
query26 1978 137 140 137
query27 3185 284 280 280
query28 7753 1845 1833 1833
query29 2181 635 593 593
query30 282 136 137 136
query31 888 694 716 694
query32 74 49 49 49
query33 702 215 207 207
query34 1188 461 460 460
query35 839 751 743 743
query36 1307 1218 1170 1170
query37 94 55 58 55
query38 3302 3228 3158 3158
query39 1343 1251 1249 1249
query40 340 86 82 82
query41 37 36 35 35
query42 86 76 80 76
query43 529 487 497 487
query44 1089 673 674 673
query45 197 183 172 172
query46 1053 641 654 641
query47 1575 1511 1518 1511
query48 387 307 304 304
query49 1214 287 277 277
query50 678 312 308 308
query51 5226 5148 5158 5148
query52 92 79 74 74
query53 322 266 263 263
query54 253 181 191 181
query55 75 76 74 74
query56 172 158 154 154
query57 1007 856 906 856
query58 181 158 158 158
query59 2730 2456 2602 2456
query60 197 175 173 173
query61 83 90 82 82
query62 628 380 349 349
query63 283 266 259 259
query64 5979 1748 1738 1738
query65 3294 3202 3223 3202
query66 1392 320 311 311
query67 14694 14304 14638 14304
query68 3728 501 490 490
query69 427 301 295 295
query70 1595 1570 1543 1543
query71 276 213 222 213
query72 3671 2835 2852 2835
query73 682 302 313 302
query74 6559 6277 6247 6247
query75 2735 2249 2270 2249
query76 2529 1018 967 967
query77 399 230 220 220
query78 9033 8761 8437 8437
query79 2966 487 483 483
query80 2044 320 313 313
query81 517 195 197 195
query82 725 84 78 78
query83 238 118 116 116
query84 304 75 74 74
query85 2375 345 336 336
query86 490 387 378 378
query87 3416 3294 3260 3260
query88 4161 2160 2148 2148
query89 410 343 363 343
query90 1968 183 180 180
query91 147 115 117 115
query92 51 41 41 41
query93 5024 453 409 409
query94 1390 157 154 154
query95 496 456 451 451
query96 635 309 306 306
query97 4256 4133 4100 4100
query98 236 193 190 190
query99 1075 700 708 700
Total cold run time: 286848 ms
Total hot run time: 173126 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913139592
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 41c40111a1ccd9485824fd01c0a23cd5cb8075c2 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913093164
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916674482
<details>
<summary>TPC-H: <b>Total hot run time: 37390 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false
------ Round 1 ----------------------------------
q1 17915 4604 4549 4549
q2 2551 142 143 142
q3 11211 947 955 947
q4 4947 855 747 747
q5 7917 2979 2904 2904
q6 192 122 125 122
q7 1180 725 714 714
q8 9294 2005 2177 2005
q9 7306 6333 6292 6292
q10 8093 2447 2444 2444
q11 419 212 205 205
q12 807 298 290 290
q13 17999 3321 3307 3307
q14 268 255 267 255
q15 517 482 486 482
q16 462 395 403 395
q17 940 544 523 523
q18 6850 6091 6026 6026
q19 1565 1361 1340 1340
q20 567 337 338 337
q21 6681 3071 3087 3071
q22 809 293 304 293
Total cold run time: 108490 ms
Total hot run time: 37390 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4493 4414 4425 4414
q2 318 229 220 220
q3 2984 2872 2844 2844
q4 1922 1708 1657 1657
q5 5185 5265 5289 5265
q6 194 114 113 113
q7 2168 1815 1781 1781
q8 3134 3233 3270 3233
q9 8394 8301 8299 8299
q10 5811 3535 3564 3535
q11 550 459 459 459
q12 757 563 569 563
q13 13882 3120 3085 3085
q14 289 248 260 248
q15 530 491 488 488
q16 499 468 457 457
q17 1855 1684 1692 1684
q18 7905 7794 7699 7699
q19 1593 1512 1555 1512
q20 2112 1947 1907 1907
q21 4857 4638 4505 4505
q22 577 486 469 469
Total cold run time: 70009 ms
Total hot run time: 54437 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916645809
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912952068
<details>
<summary>TPC-DS: <b>Total hot run time: 173064 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false
query1 949 333 321 321
query2 6570 1882 1900 1882
query3 6718 201 197 197
query4 32217 22108 21952 21952
query5 4448 418 353 353
query6 264 150 148 148
query7 4609 258 258 258
query8 238 178 173 173
query9 8563 2295 2292 2292
query10 415 204 198 198
query11 18057 15374 15300 15300
query12 130 69 65 65
query13 1711 367 372 367
query14 9111 6506 6506 6506
query15 203 178 174 174
query16 5699 255 250 250
query17 954 470 471 470
query18 1816 260 247 247
query19 175 136 133 133
query20 76 76 69 69
query21 201 138 128 128
query22 4611 4695 4681 4681
query23 31126 30438 30321 30321
query24 11903 2793 2771 2771
query25 590 302 305 302
query26 1801 139 134 134
query27 3194 285 283 283
query28 7766 1835 1820 1820
query29 1832 622 599 599
query30 276 137 137 137
query31 922 706 735 706
query32 76 49 51 49
query33 692 209 209 209
query34 1138 449 454 449
query35 885 780 771 771
query36 1307 1206 1146 1146
query37 95 61 59 59
query38 3275 3195 3154 3154
query39 1309 1264 1247 1247
query40 342 86 85 85
query41 38 36 35 35
query42 91 79 83 79
query43 514 502 478 478
query44 1093 686 686 686
query45 188 180 174 174
query46 1046 658 649 649
query47 1603 1562 1483 1483
query48 401 310 301 301
query49 1194 285 285 285
query50 696 315 304 304
query51 5305 5196 5174 5174
query52 84 85 73 73
query53 311 256 252 252
query54 249 176 195 176
query55 79 70 70 70
query56 166 157 157 157
query57 1010 915 921 915
query58 183 151 153 151
query59 2659 2513 2405 2405
query60 197 175 174 174
query61 82 83 81 81
query62 634 357 350 350
query63 265 270 268 268
query64 6032 1744 1759 1744
query65 3285 3226 3235 3226
query66 1244 319 314 314
query67 14349 14198 14287 14198
query68 3988 498 493 493
query69 428 297 293 293
query70 1447 1514 1523 1514
query71 257 211 205 205
query72 3775 2856 2799 2799
query73 697 310 305 305
query74 6755 6367 6346 6346
query75 2798 2329 2292 2292
query76 2611 1035 979 979
query77 347 233 226 226
query78 9059 8859 8605 8605
query79 2632 491 481 481
query80 1967 325 324 324
query81 515 199 195 195
query82 691 83 85 83
query83 238 117 116 116
query84 280 73 77 73
query85 2288 350 337 337
query86 519 404 367 367
query87 3541 3280 3281 3280
query88 4109 2154 2140 2140
query89 430 348 372 348
query90 1928 186 179 179
query91 165 115 117 115
query92 52 45 52 45
query93 4484 428 450 428
query94 1326 161 158 158
query95 499 455 443 443
query96 634 306 317 306
query97 4236 4122 4113 4113
query98 198 198 190 190
query99 1091 678 719 678
Total cold run time: 283247 ms
Total hot run time: 173064 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913127781
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913099244
<details>
<summary>TPC-H: <b>Total hot run time: 38064 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit caabe770ff415d2581d21754c8f962300481107c, data reload: false
------ Round 1 ----------------------------------
q1 17898 5224 5196 5196
q2 2092 145 143 143
q3 11029 981 943 943
q4 5218 731 733 731
q5 7814 2909 2962 2909
q6 191 124 119 119
q7 1179 757 724 724
q8 9339 1998 2059 1998
q9 7341 6351 6285 6285
q10 8119 2403 2397 2397
q11 421 213 200 200
q12 787 286 296 286
q13 18002 3324 3296 3296
q14 273 250 234 234
q15 522 493 484 484
q16 465 411 403 403
q17 936 579 560 560
q18 6936 6125 6119 6119
q19 1569 1396 1416 1396
q20 631 349 331 331
q21 7008 3090 3019 3019
q22 811 310 291 291
Total cold run time: 108581 ms
Total hot run time: 38064 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5151 4892 4702 4702
q2 338 245 234 234
q3 3014 2670 2725 2670
q4 1817 1578 1535 1535
q5 5425 5242 5238 5238
q6 198 115 115 115
q7 2142 1824 1779 1779
q8 3098 3184 3241 3184
q9 8650 8757 11396 8757
q10 5463 3425 3501 3425
q11 564 466 464 464
q12 786 607 615 607
q13 16939 3117 3039 3039
q14 270 243 260 243
q15 530 502 486 486
q16 512 467 468 467
q17 2094 1653 1747 1653
q18 8946 11700 8801 8801
q19 23893 1562 1499 1499
q20 4598 1918 1891 1891
q21 15415 4882 4856 4856
q22 1051 458 466 458
Total cold run time: 110894 ms
Total hot run time: 56103 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911797370
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.2 seconds inserted 10000000 Rows, about 757K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911793425
<details>
<summary>ClickBench: <b>Total hot run time: 31.01 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 882d93d0d56d3dcc3ebe3d3e9452ac2f20f1a5ca, data reload: false
query1 0.03 0.03 0.02
query2 0.06 0.03 0.03
query3 0.23 0.05 0.05
query4 1.70 0.07 0.06
query5 0.53 0.52 0.52
query6 1.27 0.64 0.63
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.55 0.49 0.53
query10 0.57 0.56 0.56
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.61 0.61 0.60
query14 0.78 0.79 0.82
query15 0.81 0.77 0.78
query16 0.37 0.38 0.40
query17 1.01 1.04 1.00
query18 0.24 0.27 0.22
query19 1.85 1.83 1.74
query20 0.02 0.01 0.01
query21 15.41 0.58 0.54
query22 2.26 2.50 2.15
query23 17.33 0.87 0.72
query24 2.32 0.72 0.92
query25 0.41 0.31 0.11
query26 0.40 0.15 0.15
query27 0.06 0.06 0.06
query28 12.03 0.78 0.77
query29 12.49 3.29 3.18
query30 0.52 0.49 0.50
query31 2.78 0.34 0.35
query32 3.37 0.47 0.48
query33 3.26 3.24 3.25
query34 15.79 4.35 4.31
query35 4.33 4.32 4.38
query36 1.13 1.10 1.10
query37 0.06 0.05 0.05
query38 0.03 0.04 0.02
query39 0.02 0.02 0.01
query40 0.19 0.13 0.14
query41 0.08 0.02 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.24 s
Total hot run time: 31.01 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913137009
<details>
<summary>TPC-DS: <b>Total hot run time: 172521 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41c40111a1ccd9485824fd01c0a23cd5cb8075c2, data reload: false
query1 948 329 324 324
query2 6586 1916 1901 1901
query3 6699 195 196 195
query4 32868 22188 22080 22080
query5 4449 346 395 346
query6 249 152 147 147
query7 4593 263 255 255
query8 243 167 176 167
query9 8597 2264 2249 2249
query10 409 208 202 202
query11 17866 15335 15389 15335
query12 114 66 68 66
query13 1670 376 383 376
query14 9227 6328 6415 6328
query15 221 175 178 175
query16 5696 251 243 243
query17 954 459 473 459
query18 1809 248 250 248
query19 181 129 131 129
query20 73 71 69 69
query21 193 129 129 129
query22 4836 4774 4631 4631
query23 31192 30546 30371 30371
query24 7397 2812 2773 2773
query25 461 313 309 309
query26 842 138 136 136
query27 2806 292 282 282
query28 4120 1845 1829 1829
query29 924 607 610 607
query30 208 136 136 136
query31 921 695 716 695
query32 77 49 50 49
query33 410 207 210 207
query34 820 444 461 444
query35 844 772 748 748
query36 1240 1222 1169 1169
query37 92 59 60 59
query38 3265 3193 3149 3149
query39 1313 1267 1253 1253
query40 194 88 80 80
query41 38 36 34 34
query42 86 77 78 77
query43 536 481 475 475
query44 1025 683 684 683
query45 190 179 176 176
query46 1028 643 625 625
query47 1631 1552 1520 1520
query48 381 299 302 299
query49 617 286 273 273
query50 665 303 298 298
query51 5313 5145 5117 5117
query52 108 73 77 73
query53 325 261 257 257
query54 225 180 178 178
query55 79 73 72 72
query56 178 158 150 150
query57 981 883 919 883
query58 178 155 143 143
query59 2761 2547 2597 2547
query60 201 175 174 174
query61 88 80 78 78
query62 453 382 354 354
query63 282 289 264 264
query64 3500 1759 1745 1745
query65 3307 3215 3217 3215
query66 1144 316 312 312
query67 14520 14248 14156 14156
query68 3816 494 487 487
query69 443 304 287 287
query70 1550 1412 1411 1411
query71 259 213 206 206
query72 3746 2841 2818 2818
query73 690 304 315 304
query74 6654 6226 6210 6210
query75 2751 2245 2293 2245
query76 2640 1029 923 923
query77 364 230 226 226
query78 9018 8744 8537 8537
query79 2830 495 493 493
query80 2015 319 311 311
query81 504 195 198 195
query82 803 87 77 77
query83 417 122 113 113
query84 294 71 83 71
query85 1360 336 312 312
query86 502 381 399 381
query87 3459 3269 3226 3226
query88 3562 2152 2143 2143
query89 401 356 354 354
query90 2006 180 179 179
query91 146 133 116 116
query92 49 42 41 41
query93 2652 420 435 420
query94 1329 157 156 156
query95 493 452 454 452
query96 596 319 315 315
query97 4259 4160 4125 4125
query98 209 200 181 181
query99 1039 731 678 678
Total cold run time: 266303 ms
Total hot run time: 172521 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913492427
<details>
<summary>TPC-H: <b>Total hot run time: 37733 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false
------ Round 1 ----------------------------------
q1 17593 4802 5161 4802
q2 2060 160 141 141
q3 10565 945 949 945
q4 6413 787 813 787
q5 7759 2988 2988 2988
q6 186 122 122 122
q7 1135 750 743 743
q8 9285 2063 2075 2063
q9 7678 6519 6317 6317
q10 8114 2452 2403 2403
q11 416 198 216 198
q12 773 281 266 266
q13 17991 3288 3348 3288
q14 272 259 246 246
q15 519 491 498 491
q16 469 395 415 395
q17 941 509 498 498
q18 6963 6115 5971 5971
q19 1571 1439 1368 1368
q20 593 345 343 343
q21 7139 3054 3137 3054
q22 819 310 304 304
Total cold run time: 109254 ms
Total hot run time: 37733 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5010 4926 4812 4812
q2 345 231 226 226
q3 3027 2773 2657 2657
q4 1835 1648 1624 1624
q5 5408 5290 5338 5290
q6 192 116 115 115
q7 2116 1794 1849 1794
q8 3126 3215 3268 3215
q9 8601 8842 8508 8508
q10 8150 3463 3514 3463
q11 572 480 480 480
q12 757 664 573 573
q13 16974 3090 3070 3070
q14 280 239 279 239
q15 527 490 496 490
q16 528 457 463 457
q17 1904 1784 1661 1661
q18 8747 8519 10477 8519
q19 15949 1560 1564 1560
q20 4607 1918 1904 1904
q21 14504 4839 4703 4703
q22 995 507 482 482
Total cold run time: 104154 ms
Total hot run time: 55842 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913140241
TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.26% (70517/249525)
Region Coverage: 27.25% (36382/133498)
Branch Coverage: 24.05% (18647/77522)
Coverage Report: http://coverage.selectdb-in.cc/coverage/41c40111a1ccd9485824fd01c0a23cd5cb8075c2_41c40111a1ccd9485824fd01c0a23cd5cb8075c2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913107611
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911975929
<details>
<summary>TPC-H: <b>Total hot run time: 38966 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c81ad8d52239a21969bd93a50037a32b916e40a8, data reload: false
------ Round 1 ----------------------------------
q1 17679 5113 5425 5113
q2 2037 139 134 134
q3 10603 950 961 950
q4 4641 778 800 778
q5 7657 2944 2926 2926
q6 180 123 120 120
q7 1084 726 721 721
q8 9236 2020 2036 2020
q9 7311 6427 6395 6395
q10 8127 2448 2480 2448
q11 411 212 205 205
q12 761 283 282 282
q13 17979 3348 3340 3340
q14 286 261 254 254
q15 558 507 501 501
q16 472 399 404 399
q17 929 619 520 520
q18 7575 6731 6686 6686
q19 1589 1366 1444 1366
q20 563 324 335 324
q21 6964 3173 3230 3173
q22 808 313 311 311
Total cold run time: 107450 ms
Total hot run time: 38966 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5497 5266 5125 5125
q2 335 212 212 212
q3 3321 3241 3144 3144
q4 2157 2097 2044 2044
q5 5973 5920 5961 5920
q6 198 120 115 115
q7 2283 1891 1880 1880
q8 3225 3365 3386 3365
q9 8961 8779 9006 8779
q10 5490 3750 3832 3750
q11 555 442 438 438
q12 803 616 630 616
q13 16899 3157 3116 3116
q14 267 254 270 254
q15 555 510 500 500
q16 519 483 458 458
q17 1862 1847 1872 1847
q18 9505 19832 9643 9643
q19 24515 1598 1518 1518
q20 4671 1927 1906 1906
q21 14632 5647 5643 5643
q22 1006 539 565 539
Total cold run time: 113229 ms
Total hot run time: 60812 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913093762
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1471069602
##########
be/src/http/action/stream_load.cpp:
##########
@@ -350,7 +354,15 @@ void StreamLoadAction::on_chunk_data(HttpRequest* req) {
}
ctx->receive_bytes += remove_bytes;
}
- ctx->read_data_cost_nanos += (MonotonicNanos() - start_read_data_time);
+ int64_t read_data_time = MonotonicNanos() - start_read_data_time;
+ int64_t last_receive_and_read_data_cost_nanos = ctx->receive_and_read_data_cost_nanos;
+ ctx->read_data_cost_nanos += read_data_time;
+ ctx->receive_and_read_data_cost_nanos =
Review Comment:
please add a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on code in PR #30432:
URL: https://github.com/apache/doris/pull/30432#discussion_r1467562899
##########
be/src/runtime/stream_load/stream_load_executor.cpp:
##########
@@ -148,6 +148,13 @@ Status StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<StreamLoadConte
static_cast<void>(this->commit_txn(ctx.get()));
}
}
+
+ LOG(INFO) << "finished to execute job. label=" << ctx->label << ", txn_id=" << ctx->txn_id
Review Comment:
> leave a keyword such as streamload in the log.
I will update it.
> and consider adding a bvar latency recorder for streamload
Perhaps bvar needs to be on-site, but without it, we may not know what happened?
Maybe both log and bvar are needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913124693
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911739745
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1911734061
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913508532
<details>
<summary>ClickBench: <b>Total hot run time: 30.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b4e7848cc055858ccb3f0249195afba7a880276, data reload: false
query1 0.04 0.03 0.04
query2 0.06 0.02 0.02
query3 0.22 0.06 0.05
query4 1.67 0.10 0.09
query5 0.54 0.51 0.51
query6 1.19 0.64 0.65
query7 0.01 0.01 0.02
query8 0.04 0.02 0.02
query9 0.54 0.51 0.49
query10 0.55 0.55 0.54
query11 0.11 0.09 0.08
query12 0.11 0.09 0.09
query13 0.60 0.60 0.62
query14 0.81 0.79 0.78
query15 0.79 0.78 0.79
query16 0.38 0.39 0.39
query17 1.03 1.04 1.01
query18 0.22 0.26 0.26
query19 1.80 1.69 1.76
query20 0.02 0.01 0.01
query21 15.41 0.58 0.58
query22 2.45 2.19 1.58
query23 17.46 0.83 0.82
query24 2.44 0.78 1.29
query25 0.27 0.29 0.10
query26 0.56 0.14 0.14
query27 0.05 0.05 0.06
query28 11.13 0.85 0.84
query29 12.58 3.30 3.08
query30 0.68 0.55 0.55
query31 2.79 0.34 0.36
query32 3.36 0.48 0.49
query33 3.20 3.18 3.25
query34 15.76 4.30 4.31
query35 4.34 4.40 4.44
query36 1.08 1.06 1.07
query37 0.06 0.05 0.05
query38 0.03 0.02 0.03
query39 0.02 0.01 0.02
query40 0.16 0.12 0.15
query41 0.07 0.02 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.67 s
Total hot run time: 30.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913104296
<details>
<summary>ClickBench: <b>Total hot run time: 30.11 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit caabe770ff415d2581d21754c8f962300481107c, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.09 0.10
query5 0.53 0.51 0.52
query6 1.24 0.62 0.63
query7 0.02 0.02 0.01
query8 0.04 0.02 0.02
query9 0.54 0.50 0.51
query10 0.57 0.56 0.55
query11 0.12 0.08 0.08
query12 0.10 0.09 0.10
query13 0.62 0.60 0.60
query14 0.77 0.79 0.80
query15 0.78 0.79 0.78
query16 0.38 0.39 0.39
query17 1.06 1.00 1.01
query18 0.22 0.27 0.22
query19 1.77 1.80 1.74
query20 0.01 0.00 0.01
query21 15.42 0.55 0.56
query22 2.82 2.08 1.64
query23 17.35 0.72 0.79
query24 2.36 1.34 0.44
query25 0.25 0.32 0.04
query26 0.50 0.14 0.13
query27 0.06 0.05 0.05
query28 12.10 0.83 0.85
query29 12.59 3.25 3.16
query30 0.59 0.54 0.54
query31 2.79 0.36 0.35
query32 3.36 0.48 0.49
query33 3.19 3.26 3.24
query34 15.58 4.24 4.23
query35 4.39 4.28 4.39
query36 1.10 1.07 1.07
query37 0.06 0.05 0.06
query38 0.04 0.03 0.03
query39 0.02 0.01 0.02
query40 0.18 0.13 0.14
query41 0.08 0.01 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.63 s
Total hot run time: 30.11 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1913105034
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit caabe770ff415d2581d21754c8f962300481107c with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.1 seconds inserted 10000000 Rows, about 709K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1912954272
<details>
<summary>ClickBench: <b>Total hot run time: 31.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cbcb6998621b798b06ec737f0a296f186841b677, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.22 0.06 0.06
query4 1.68 0.09 0.10
query5 0.52 0.51 0.52
query6 1.19 0.62 0.62
query7 0.02 0.02 0.01
query8 0.04 0.03 0.02
query9 0.54 0.50 0.50
query10 0.55 0.57 0.54
query11 0.12 0.08 0.08
query12 0.11 0.09 0.09
query13 0.60 0.60 0.60
query14 0.77 0.80 0.81
query15 0.79 0.77 0.80
query16 0.37 0.39 0.38
query17 1.01 0.99 1.03
query18 0.21 0.25 0.25
query19 1.79 1.80 1.78
query20 0.01 0.01 0.01
query21 15.41 0.58 0.59
query22 2.51 2.86 2.08
query23 17.36 0.83 0.84
query24 2.36 0.70 1.29
query25 0.26 0.31 0.17
query26 0.47 0.13 0.13
query27 0.05 0.04 0.05
query28 11.66 0.83 0.87
query29 12.56 3.17 3.24
query30 0.66 0.54 0.55
query31 2.78 0.36 0.35
query32 3.36 0.47 0.48
query33 3.22 3.18 3.20
query34 15.97 4.41 4.37
query35 4.43 4.31 4.24
query36 1.11 1.06 1.06
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.02
query40 0.17 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.22 s
Total hot run time: 31.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916678355
TeamCity be ut coverage result:
Function Coverage: 36.18% (8624/23835)
Line Coverage: 28.24% (70506/249695)
Region Coverage: 27.24% (36383/133580)
Branch Coverage: 24.02% (18631/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd_3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916655310
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](stream-load) record time on receiving HTTP request [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30432:
URL: https://github.com/apache/doris/pull/30432#issuecomment-1916700249
<details>
<summary>ClickBench: <b>Total hot run time: 30.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c2af3aadc12ae1e61bcedb2cf68caadbd6fe1bd, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.68 0.10 0.11
query5 0.54 0.51 0.52
query6 1.24 0.65 0.64
query7 0.02 0.01 0.02
query8 0.03 0.03 0.02
query9 0.55 0.51 0.50
query10 0.54 0.54 0.56
query11 0.11 0.08 0.08
query12 0.11 0.09 0.09
query13 0.60 0.62 0.60
query14 0.79 0.80 0.79
query15 0.80 0.78 0.76
query16 0.39 0.37 0.39
query17 1.00 1.05 1.06
query18 0.20 0.28 0.23
query19 1.89 1.80 1.77
query20 0.01 0.01 0.01
query21 15.41 0.60 0.57
query22 2.43 2.10 1.55
query23 17.38 0.88 0.91
query24 2.23 1.08 0.46
query25 0.36 0.18 0.15
query26 0.39 0.13 0.14
query27 0.05 0.05 0.05
query28 12.16 0.84 0.82
query29 12.56 3.15 3.11
query30 0.62 0.55 0.53
query31 2.78 0.34 0.35
query32 3.37 0.48 0.48
query33 3.22 3.20 3.21
query34 15.74 4.29 4.38
query35 4.29 4.27 4.33
query36 1.10 1.05 1.05
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.16 0.12 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.3 s
Total hot run time: 30.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org