You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2024/07/11 08:35:09 UTC
[PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
ByteYue opened a new pull request, #37663:
URL: https://github.com/apache/doris/pull/37663
## Proposed changes
<!--Describe your changes.-->
We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #37663:
URL: https://github.com/apache/doris/pull/37663#discussion_r1677035120
##########
common/cpp/s3_rate_limiter.cpp:
##########
@@ -17,22 +17,49 @@
#include "s3_rate_limiter.h"
-#include <fmt/format.h>
-
#include <chrono>
+#include <mutex>
#include <thread>
-namespace doris::cloud {
+namespace doris {
// Just 10^6.
static constexpr auto MS = 1000000UL;
+class S3RateLimiter::SimpleSpinLock {
+public:
+ SimpleSpinLock() = default;
+ ~SimpleSpinLock() = default;
+
+ void lock() {
+ while (_flag.test_and_set(std::memory_order_acq_rel)) {
+ // Spin until we acquire the lock
Review Comment:
Log if too many times tried?
##########
common/cpp/s3_rate_limiter.cpp:
##########
Review Comment:
Are there any tests for rate limiter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227697084
<details>
<summary>TPC-H: <b>Total hot run time: 40327 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false
------ Round 1 ----------------------------------
q1 17823 4528 4384 4384
q2 2545 201 195 195
q3 11792 1195 1190 1190
q4 10306 800 738 738
q5 7598 2750 2778 2750
q6 226 147 150 147
q7 971 621 612 612
q8 9480 2110 2110 2110
q9 8789 6606 6599 6599
q10 8835 3830 3756 3756
q11 448 238 239 238
q12 395 231 232 231
q13 17868 2993 2997 2993
q14 289 238 239 238
q15 530 495 488 488
q16 508 384 380 380
q17 995 646 663 646
q18 8247 7503 7403 7403
q19 6791 1422 1463 1422
q20 704 334 341 334
q21 4930 3181 3317 3181
q22 371 292 293 292
Total cold run time: 120441 ms
Total hot run time: 40327 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4346 4261 4285 4261
q2 375 276 266 266
q3 3092 2760 2769 2760
q4 1905 1567 1634 1567
q5 5304 5322 5349 5322
q6 220 134 134 134
q7 2169 1707 1738 1707
q8 3209 3395 3346 3346
q9 8503 8413 8435 8413
q10 3909 3727 3743 3727
q11 607 487 488 487
q12 775 624 606 606
q13 16903 3009 3006 3006
q14 299 271 275 271
q15 523 487 494 487
q16 491 422 435 422
q17 1773 1502 1456 1456
q18 7669 7512 7396 7396
q19 1712 1611 1449 1449
q20 2015 1828 1801 1801
q21 4846 4822 4647 4647
q22 575 495 494 494
Total cold run time: 71220 ms
Total hot run time: 54025 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37663:
URL: https://github.com/apache/doris/pull/37663#discussion_r1677016261
##########
cloud/test/s3_rate_limiter_test.cpp:
##########
@@ -15,8 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#include <cpp/s3_rate_limiter.h>
Review Comment:
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
```cpp
#include <gtest/gtest.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2226958270
> BE may use the same rate limiter?
Yes. I'll add it into BE later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227648928
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227705246
<details>
<summary>TPC-DS: <b>Total hot run time: 173782 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false
query1 910 365 367 365
query2 6452 1885 1864 1864
query3 6672 210 211 210
query4 28266 17649 17403 17403
query5 4180 521 490 490
query6 255 181 165 165
query7 4594 299 288 288
query8 242 197 183 183
query9 8590 2466 2427 2427
query10 446 309 274 274
query11 11715 10114 10041 10041
query12 130 85 80 80
query13 1626 362 375 362
query14 10388 7809 7819 7809
query15 220 171 167 167
query16 7605 314 315 314
query17 1742 536 533 533
query18 1255 274 275 274
query19 192 150 150 150
query20 91 84 83 83
query21 210 130 120 120
query22 4364 4102 3946 3946
query23 33776 33206 33176 33176
query24 11991 2841 2848 2841
query25 633 365 363 363
query26 1637 150 150 150
query27 2918 277 274 274
query28 7703 2044 2062 2044
query29 1026 627 607 607
query30 283 149 148 148
query31 951 743 760 743
query32 96 53 55 53
query33 780 296 290 290
query34 967 492 492 492
query35 695 609 568 568
query36 1130 944 944 944
query37 153 81 83 81
query38 2886 2720 2730 2720
query39 884 814 808 808
query40 279 132 127 127
query41 54 50 47 47
query42 122 96 107 96
query43 505 470 479 470
query44 1199 755 737 737
query45 194 167 163 163
query46 1092 763 736 736
query47 1857 1772 1783 1772
query48 375 293 298 293
query49 1211 445 431 431
query50 803 402 402 402
query51 6865 6860 6795 6795
query52 104 96 101 96
query53 357 297 311 297
query54 1050 462 474 462
query55 80 75 78 75
query56 305 296 288 288
query57 1161 1059 1078 1059
query58 274 249 267 249
query59 3037 2600 2564 2564
query60 333 299 304 299
query61 116 114 113 113
query62 825 664 659 659
query63 334 301 293 293
query64 10665 2308 1807 1807
query65 3318 3120 3113 3113
query66 1286 346 368 346
query67 15757 15128 15020 15020
query68 4647 566 562 562
query69 624 466 358 358
query70 1232 1156 1052 1052
query71 436 296 289 289
query72 8578 6208 5647 5647
query73 758 342 327 327
query74 6156 5640 5727 5640
query75 3719 2741 2674 2674
query76 3036 921 960 921
query77 674 328 321 321
query78 9623 9113 9053 9053
query79 9109 537 536 536
query80 1858 520 495 495
query81 583 225 222 222
query82 1631 139 134 134
query83 358 169 174 169
query84 275 83 89 83
query85 2049 308 298 298
query86 446 334 320 320
query87 3332 3078 3114 3078
query88 5568 2393 2393 2393
query89 527 405 386 386
query90 1813 198 197 197
query91 133 99 104 99
query92 63 51 52 51
query93 6715 527 519 519
query94 1097 219 225 219
query95 414 322 333 322
query96 620 273 279 273
query97 3194 3086 3041 3041
query98 232 204 196 196
query99 1562 1248 1267 1248
Total cold run time: 303274 ms
Total hot run time: 173782 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2222351018
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on code in PR #37663:
URL: https://github.com/apache/doris/pull/37663#discussion_r1677281957
##########
common/cpp/s3_rate_limiter.cpp:
##########
Review Comment:
The ut lies in cloud/test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #37663:
URL: https://github.com/apache/doris/pull/37663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227207632
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2222361342
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227709350
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.67 0.07 0.08
query5 0.50 0.49 0.50
query6 1.14 0.74 0.73
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.55 0.50 0.48
query10 0.54 0.52 0.55
query11 0.15 0.11 0.11
query12 0.15 0.13 0.13
query13 0.60 0.59 0.59
query14 0.79 0.77 0.79
query15 0.86 0.82 0.81
query16 0.34 0.35 0.37
query17 0.98 0.98 1.01
query18 0.24 0.23 0.23
query19 1.86 1.74 1.72
query20 0.02 0.01 0.01
query21 15.41 0.73 0.65
query22 4.00 7.53 1.88
query23 18.25 1.26 1.21
query24 2.10 0.23 0.23
query25 0.15 0.09 0.09
query26 0.28 0.20 0.20
query27 0.46 0.23 0.23
query28 13.20 1.02 1.01
query29 12.58 3.36 3.35
query30 0.25 0.06 0.05
query31 2.87 0.39 0.38
query32 3.28 0.49 0.47
query33 2.90 2.85 2.94
query34 16.88 4.35 4.37
query35 4.43 4.44 4.40
query36 0.66 0.46 0.50
query37 0.20 0.15 0.15
query38 0.15 0.15 0.14
query39 0.05 0.04 0.03
query40 0.15 0.12 0.11
query41 0.09 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.25 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2230271277
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2230271182
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2222354255
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37663:
URL: https://github.com/apache/doris/pull/37663#discussion_r1677016731
##########
cloud/test/s3_rate_limiter_test.cpp:
##########
@@ -15,8 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#include <cpp/s3_rate_limiter.h>
#include <gtest/gtest.h>
Review Comment:
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
```cpp
#include <gtest/gtest.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](Recycler) Add rate limit for all obj storage client implementation [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #37663:
URL: https://github.com/apache/doris/pull/37663#issuecomment-2227397777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org