You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2024/07/24 06:57:26 UTC
[PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
ByteYue opened a new pull request, #38294:
URL: https://github.com/apache/doris/pull/38294
## Proposed changes
<!--Describe your changes.-->
Add one bvar to monitor the failed request due to exceeds rate limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256128530
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256190404
<details>
<summary>TPC-H: <b>Total hot run time: 41956 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ecd971015410c28066b4d41c1d83c0033adbce1, data reload: false
------ Round 1 ----------------------------------
q1 17620 4181 4051 4051
q2 2017 204 213 204
q3 10605 1372 1365 1365
q4 10700 841 1022 841
q5 8262 3140 3054 3054
q6 225 138 137 137
q7 1061 626 631 626
q8 10438 1884 2012 1884
q9 8788 6759 6678 6678
q10 8759 3829 3858 3829
q11 438 256 251 251
q12 414 224 228 224
q13 17763 2920 2925 2920
q14 273 251 248 248
q15 526 482 484 482
q16 513 397 397 397
q17 977 929 955 929
q18 8143 7491 7297 7297
q19 1546 1221 1227 1221
q20 573 322 357 322
q21 5349 4717 4805 4717
q22 359 282 279 279
Total cold run time: 115349 ms
Total hot run time: 41956 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4095 4014 4026 4014
q2 332 218 222 218
q3 3001 3033 2974 2974
q4 1905 1892 1889 1889
q5 5277 5252 5233 5233
q6 217 128 127 127
q7 2060 1699 1672 1672
q8 3223 3324 3312 3312
q9 8344 8382 8356 8356
q10 3779 3858 3908 3858
q11 560 449 442 442
q12 713 555 546 546
q13 9777 2954 2914 2914
q14 283 255 257 255
q15 517 469 473 469
q16 442 395 397 395
q17 1732 1684 1649 1649
q18 7676 7487 7258 7258
q19 1671 1691 1676 1676
q20 1969 1763 1736 1736
q21 5485 5132 5281 5132
q22 509 430 457 430
Total cold run time: 63567 ms
Total hot run time: 54555 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #38294:
URL: https://github.com/apache/doris/pull/38294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2247060339
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256270709
<details>
<summary>TPC-DS: <b>Total hot run time: 168590 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false
query1 941 373 369 369
query2 6479 1676 1665 1665
query3 6667 207 221 207
query4 20372 17445 17282 17282
query5 4325 525 512 512
query6 285 167 173 167
query7 4595 306 299 299
query8 258 193 201 193
query9 8508 2434 2391 2391
query10 439 271 255 255
query11 10486 10081 9978 9978
query12 134 89 87 87
query13 1632 373 368 368
query14 9605 7018 8478 7018
query15 200 173 160 160
query16 7086 488 450 450
query17 945 604 550 550
query18 1914 285 275 275
query19 195 144 142 142
query20 89 86 86 86
query21 206 103 105 103
query22 4333 3981 3920 3920
query23 33762 33169 32843 32843
query24 10382 3084 2998 2998
query25 685 385 391 385
query26 1802 153 153 153
query27 2996 280 279 279
query28 7074 2007 1984 1984
query29 1345 444 452 444
query30 280 151 151 151
query31 925 751 738 738
query32 103 59 57 57
query33 700 310 309 309
query34 910 480 482 480
query35 845 710 733 710
query36 1004 865 869 865
query37 312 81 80 80
query38 2859 2737 2808 2737
query39 846 812 818 812
query40 283 114 117 114
query41 48 45 48 45
query42 121 99 99 99
query43 473 397 400 397
query44 1190 717 720 717
query45 213 177 174 174
query46 1088 787 774 774
query47 1821 1725 1735 1725
query48 375 292 294 292
query49 1201 440 442 440
query50 898 434 472 434
query51 6709 6595 6664 6595
query52 103 96 90 90
query53 256 179 175 175
query54 617 452 453 452
query55 77 75 79 75
query56 282 258 254 254
query57 1136 1076 1025 1025
query58 281 261 257 257
query59 2627 2487 2280 2280
query60 301 266 283 266
query61 101 97 98 97
query62 923 658 682 658
query63 210 186 199 186
query64 5946 1914 1912 1912
query65 3146 3119 3093 3093
query66 1444 333 335 333
query67 15338 14860 14851 14851
query68 4341 562 575 562
query69 450 300 306 300
query70 1128 1025 1046 1025
query71 359 269 281 269
query72 7181 2691 2542 2542
query73 760 327 338 327
query74 6047 5631 5613 5613
query75 3343 2737 2711 2711
query76 2245 1361 1414 1361
query77 437 305 300 300
query78 9502 8979 8865 8865
query79 1649 539 524 524
query80 1201 510 496 496
query81 571 227 228 227
query82 1163 134 128 128
query83 237 169 164 164
query84 274 78 81 78
query85 1282 321 303 303
query86 394 279 310 279
query87 3259 3150 3122 3122
query88 2962 2495 2530 2495
query89 378 289 284 284
query90 1760 191 189 189
query91 133 102 104 102
query92 62 51 52 51
query93 1456 613 614 613
query94 887 305 302 302
query95 384 268 270 268
query96 601 281 280 280
query97 3161 3028 3036 3028
query98 222 204 197 197
query99 1673 1310 1273 1273
Total cold run time: 263234 ms
Total hot run time: 168590 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #38294:
URL: https://github.com/apache/doris/pull/38294#discussion_r1690979539
##########
cloud/src/recycler/s3_accessor.h:
##########
@@ -17,10 +17,13 @@
#pragma once
+#include <bvar/latency_recorder.h>
Review Comment:
warning: 'bvar/latency_recorder.h' file not found [clang-diagnostic-error]
```cpp
#include <bvar/latency_recorder.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #38294:
URL: https://github.com/apache/doris/pull/38294#discussion_r1695354419
##########
common/cpp/s3_rate_limiter.cpp:
##########
@@ -69,29 +70,31 @@ S3RateLimiter::~S3RateLimiter() = default;
S3RateLimiterHolder::~S3RateLimiterHolder() = default;
-std::pair<size_t, double> S3RateLimiter::_update_remain_token(
- std::chrono::system_clock::time_point now, size_t amount) {
+std::pair<size_t, double> S3RateLimiter::_update_remain_token(long now, size_t amount) {
// Values obtained under lock to be checked after release
size_t count_value;
double tokens_value;
{
std::lock_guard<SimpleSpinLock> lock(*_mutex);
if (_max_speed) {
- double delta_seconds = static_cast<double>((now - _prev_ms).count()) / MS;
+ double delta_seconds =
+ _prev_ns_count ? static_cast<double>(now - _prev_ns_count) / NS : 0;
_remain_tokens = std::min<double>(_remain_tokens + _max_speed * delta_seconds - amount,
_max_burst);
}
_count += amount;
count_value = _count;
tokens_value = _remain_tokens;
- _prev_ms = now;
+ _prev_ns_count = now;
}
return {count_value, tokens_value};
}
int64_t S3RateLimiter::add(size_t amount) {
// Values obtained under lock to be checked after release
- auto [count_value, tokens_value] = _update_remain_token(CURRENT_TIME, amount);
+ auto time = CURRENT_TIME;
+ auto [count_value, tokens_value] =
+ _update_remain_token(time.time_since_epoch().count(), amount);
Review Comment:
explicit duration cast as nano
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256128448
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256079386
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256207635
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2247828897
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2247630793
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor OSS's rate limit sleep time and failure time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2247050599
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256282548
<details>
<summary>ClickBench: <b>Total hot run time: 30.18 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.03 0.03
query3 0.21 0.05 0.05
query4 1.68 0.07 0.07
query5 0.49 0.50 0.48
query6 1.13 0.72 0.71
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.58 0.53 0.50
query10 0.57 0.56 0.56
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.61 0.60 0.59
query14 0.77 0.78 0.82
query15 0.88 0.85 0.86
query16 0.36 0.35 0.36
query17 0.99 0.97 1.00
query18 0.22 0.21 0.21
query19 1.84 1.74 1.76
query20 0.01 0.01 0.01
query21 15.43 0.74 0.67
query22 3.78 6.70 1.51
query23 17.97 1.37 1.31
query24 2.31 0.23 0.23
query25 0.18 0.09 0.09
query26 0.31 0.21 0.22
query27 0.46 0.23 0.23
query28 13.17 0.99 0.97
query29 12.52 3.25 3.25
query30 0.25 0.05 0.05
query31 2.86 0.41 0.40
query32 3.24 0.51 0.48
query33 2.97 2.93 2.93
query34 15.44 4.26 4.26
query35 4.32 4.30 4.29
query36 0.67 0.48 0.48
query37 0.19 0.17 0.16
query38 0.17 0.14 0.14
query39 0.04 0.04 0.03
query40 0.16 0.14 0.14
query41 0.09 0.05 0.05
query42 0.05 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 107.44 s
Total hot run time: 30.18 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2258468550
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](RateLimit) Add bvar to monitor object storage rate limit sleep time and failure time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38294:
URL: https://github.com/apache/doris/pull/38294#issuecomment-2256247675
<details>
<summary>TPC-H: <b>Total hot run time: 41917 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ac14c9f980c5e68dd0f0a24e3250d6030e6aff0, data reload: false
------ Round 1 ----------------------------------
q1 17629 4131 4120 4120
q2 2030 207 214 207
q3 10452 1251 1379 1251
q4 10157 861 961 861
q5 7659 3024 3013 3013
q6 226 141 140 140
q7 1039 619 638 619
q8 9472 1884 1943 1884
q9 9082 6634 6642 6634
q10 8805 3892 3891 3891
q11 439 256 256 256
q12 449 229 235 229
q13 17755 2923 2963 2923
q14 270 247 250 247
q15 530 488 501 488
q16 524 399 391 391
q17 986 962 906 906
q18 8032 7367 7247 7247
q19 1375 1227 1210 1210
q20 570 338 335 335
q21 5411 4782 4855 4782
q22 356 285 283 283
Total cold run time: 113248 ms
Total hot run time: 41917 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4080 4031 4023 4023
q2 327 221 220 220
q3 3014 2981 2973 2973
q4 1933 1904 1936 1904
q5 5247 5268 5261 5261
q6 223 133 130 130
q7 2105 1665 1704 1665
q8 3227 3332 3349 3332
q9 8332 8341 8327 8327
q10 3799 3871 3845 3845
q11 555 452 451 451
q12 721 550 548 548
q13 5125 2975 2987 2975
q14 299 266 259 259
q15 515 473 471 471
q16 471 406 395 395
q17 1721 1715 1701 1701
q18 7754 7416 7203 7203
q19 1688 1693 1694 1693
q20 1973 1789 1764 1764
q21 5503 5403 5216 5216
q22 505 477 463 463
Total cold run time: 59117 ms
Total hot run time: 54819 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org