You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2023/11/28 15:20:18 UTC
[PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Yukang-Lian opened a new pull request, #27726:
URL: https://github.com/apache/doris/pull/27726
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
If available bytes >= 1G, max wal disk limit is 5% of available bytes.
If 1G > available bytes >= 128M, max wal disk limit is 64M.
If available bytes < 128M, max wal disk limit is 10% of available bytes.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1432189587
##########
be/src/vec/sink/group_commit_block_sink.h:
##########
@@ -75,6 +77,7 @@ class GroupCommitBlockSink : public DataSink {
std::vector<VOlapTablePartition*> _partitions;
Bitmap _filter_bitmap;
bool _has_filtered_rows = false;
+ std::mutex lock;
Review Comment:
is this lock unused?
##########
be/src/vec/sink/group_commit_block_sink.h:
##########
@@ -47,7 +47,9 @@ class GroupCommitBlockSink : public DataSink {
private:
Status _add_block(RuntimeState* state, std::shared_ptr<vectorized::Block> block);
- Status _add_blocks();
+ Status _add_blocks(bool write_wal);
+ const string& _get_max_disk_usage_dir(
Review Comment:
is this function unused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868301669
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.19 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183750098 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869611444
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858773147
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17224125587 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859004927
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862162592
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1421413914
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +123,92 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:126:** 86 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848609994
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 7e84258236e3dbd3954f27f7ff17e58b2a8c485e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4694 4440 4471 4440
q2 360 143 158 143
q3 1466 1257 1223 1223
q4 1128 933 942 933
q5 3141 3163 3167 3163
q6 253 129 129 129
q7 1011 506 478 478
q8 2182 2240 2178 2178
q9 6733 6655 6670 6655
q10 3224 3238 3274 3238
q11 331 208 204 204
q12 359 210 212 210
q13 4600 3847 3843 3843
q14 242 213 217 213
q15 559 517 523 517
q16 446 386 395 386
q17 1001 558 588 558
q18 8247 6949 7007 6949
q19 1510 1308 1449 1308
q20 533 334 319 319
q21 3093 2704 2669 2669
q22 352 291 294 291
Total cold run time: 45465 ms
Total hot run time: 40047 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4624 4405 4395 4395
q2 268 165 176 165
q3 3573 3551 3557 3551
q4 2391 2393 2391 2391
q5 5761 5758 5743 5743
q6 240 121 122 121
q7 2423 1886 1869 1869
q8 3526 3518 3522 3518
q9 9045 9038 9051 9038
q10 3891 4003 3999 3999
q11 503 383 370 370
q12 762 587 596 587
q13 4333 3548 3552 3548
q14 283 255 254 254
q15 575 523 521 521
q16 503 470 476 470
q17 1886 1863 1908 1863
q18 8793 8359 8360 8359
q19 1732 1756 1729 1729
q20 2277 1966 1939 1939
q21 6605 6226 6165 6165
q22 499 429 432 429
Total cold run time: 64493 ms
Total hot run time: 61024 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1421394889
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +122,94 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:125:** 88 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1435051920
##########
be/src/runtime/group_commit_mgr.h:
##########
@@ -144,7 +152,11 @@ class GroupCommitMgr {
Status get_first_block_load_queue(int64_t db_id, int64_t table_id, int64_t base_schema_version,
const UniqueId& load_id,
std::shared_ptr<LoadBlockQueue>& load_block_queue,
- int be_exe_version);
+ int be_exe_version, bool write_wal);
Review Comment:
the `bool writer_wal` is unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868218071
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit fc0ad82d51128abca2df68d01cf2c7de8d877d40, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4727 4390 4440 4390
q2 363 155 158 155
q3 1469 1306 1249 1249
q4 1099 935 918 918
q5 3200 3165 3173 3165
q6 248 133 132 132
q7 975 485 489 485
q8 2188 2213 2208 2208
q9 6649 6665 6684 6665
q10 3231 3285 3296 3285
q11 309 180 193 180
q12 354 208 212 208
q13 4572 3836 3787 3787
q14 249 217 217 217
q15 567 528 525 525
q16 445 381 392 381
q17 1006 682 571 571
q18 7170 6940 6916 6916
q19 1540 1415 1432 1415
q20 515 320 314 314
q21 3098 2606 2640 2606
q22 354 285 285 285
Total cold run time: 44328 ms
Total hot run time: 40057 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4313 4303 4354 4303
q2 273 168 166 166
q3 3494 3509 3495 3495
q4 2384 2359 2369 2359
q5 5700 5720 5735 5720
q6 242 122 126 122
q7 2393 1862 1873 1862
q8 3520 3521 3508 3508
q9 9009 8953 8983 8953
q10 3915 4024 4022 4022
q11 491 365 369 365
q12 770 599 604 599
q13 4293 3595 3524 3524
q14 288 263 256 256
q15 568 516 523 516
q16 507 462 475 462
q17 1895 1839 1848 1839
q18 8569 8158 8022 8022
q19 1729 1770 1761 1761
q20 2248 1949 1928 1928
q21 6485 6121 6130 6121
q22 507 434 423 423
Total cold run time: 63593 ms
Total hot run time: 60326 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1410294859
##########
be/src/olap/wal_manager.cpp:
##########
@@ -75,11 +78,42 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ RETURN_IF_ERROR(init_wal_limit());
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_limit() {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ // Get the root path available space.
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info("./", &disk_capacity_bytes,
+ &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit =
+ ParseUtil::parse_mem_spec(config::wal_max_disk_limit, -1, available_bytes, &is_percent);
+ if (wal_disk_limit < 0 || wal_disk_limit > available_bytes) {
+ return Status::InternalError(
+ "wal_max_disk_limit config is wrong, please check your config!");
+ }
+ WalManager::wal_limit = wal_disk_limit;
+ return Status::OK();
+}
+
+// Just used for ut.
+Status WalManager::init_wal_limit(size_t available_bytes) {
Review Comment:
warning: method 'init_wal_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:77:
```diff
- Status init_wal_limit(size_t available_bytes);
+ static Status init_wal_limit(size_t available_bytes);
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +124,91 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:127:** 85 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1835452338
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1830092107
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1830352657
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b6418cc746d1c4dda484e892da9cfd46a68ff315, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4941 4664 4670 4664
q2 351 174 153 153
q3 1511 1296 1292 1292
q4 1159 1043 957 957
q5 3262 3267 3224 3224
q6 255 134 133 133
q7 1036 518 550 518
q8 2231 2241 2221 2221
q9 6967 6956 6933 6933
q10 3295 3365 3354 3354
q11 334 212 209 209
q12 352 217 224 217
q13 4632 3911 3861 3861
q14 250 219 218 218
q15 587 534 519 519
q16 423 404 399 399
q17 1038 651 642 642
q18 7938 7406 7406 7406
q19 1551 1566 1532 1532
q20 562 314 296 296
q21 3401 2992 2965 2965
q22 368 295 304 295
Total cold run time: 46444 ms
Total hot run time: 42008 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4601 4594 4578 4578
q2 315 191 228 191
q3 3748 3749 3745 3745
q4 2516 2518 2495 2495
q5 6181 6186 6175 6175
q6 245 126 127 126
q7 2668 2002 1947 1947
q8 3681 3716 3700 3700
q9 9377 9287 9366 9287
q10 4060 4144 4149 4144
q11 670 492 494 492
q12 812 632 617 617
q13 4363 3630 3597 3597
q14 291 250 246 246
q15 596 521 528 521
q16 524 493 486 486
q17 2111 2052 2076 2052
q18 9552 8925 9021 8925
q19 1811 1770 1758 1758
q20 2305 1981 1980 1980
q21 7216 7152 7176 7152
q22 644 545 555 545
Total cold run time: 68287 ms
Total hot run time: 64759 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831128164
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848586170
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859035953
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 6cddf0bc248284ed728e18295dcdf5c9c33e7da2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4764 4463 4509 4463
q2 363 151 159 151
q3 1472 1263 1198 1198
q4 1115 914 913 913
q5 3168 3157 3200 3157
q6 245 126 128 126
q7 993 490 484 484
q8 2175 2194 2191 2191
q9 6698 6669 6662 6662
q10 3218 3281 3267 3267
q11 318 211 206 206
q12 354 206 205 205
q13 4554 3824 3793 3793
q14 240 211 211 211
q15 566 522 516 516
q16 437 376 390 376
q17 1021 711 565 565
q18 7204 6832 6989 6832
q19 1557 1481 1401 1401
q20 551 308 307 307
q21 3087 2666 2692 2666
q22 355 282 283 282
Total cold run time: 44455 ms
Total hot run time: 39972 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4422 4410 4406 4406
q2 265 164 171 164
q3 3550 3538 3516 3516
q4 2407 2388 2380 2380
q5 5738 5718 5726 5718
q6 242 119 122 119
q7 2384 1894 1880 1880
q8 3529 3526 3539 3526
q9 9031 9009 9007 9007
q10 3898 4009 4008 4008
q11 515 377 377 377
q12 762 616 593 593
q13 4285 3572 3571 3571
q14 283 249 262 249
q15 575 528 532 528
q16 497 440 461 440
q17 1872 1875 1835 1835
q18 8667 8082 8291 8082
q19 1714 1752 1727 1727
q20 2261 1941 1926 1926
q21 6509 6149 6192 6149
q22 506 418 415 415
Total cold run time: 63912 ms
Total hot run time: 60616 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858763699
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1857301944
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859050686
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859030759
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1431168907
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -257,19 +263,20 @@ Status GroupCommitTable::_create_group_commit_load(
params.desc_tbl.slotDescriptors, be_exe_version));
_cv.notify_all();
}
- st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params,
- pipeline_params);
+ st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params, pipeline_params,
+ write_wal);
if (!st.ok()) {
static_cast<void>(_finish_group_commit_load(_db_id, _table_id, label, txn_id, instance_id,
- st, true, nullptr));
+ st, true, nullptr, write_wal));
}
return st;
}
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
Review Comment:
warning: function '_finish_group_commit_load' has cognitive complexity of 74 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/group_commit_mgr.cpp:281:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:288:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (state) {
^
```
**be/src/runtime/group_commit_mgr.cpp:295:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:300:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:311:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:321:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (it != _load_block_queues.end()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:324:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:324:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:325:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:325:** +1
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:336:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!st.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:342:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:344:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:344:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:345:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:345:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:350:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:351:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:351:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:352:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:352:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:357:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!prepare_failed && !result_status.ok() &&
^
```
**be/src/runtime/group_commit_mgr.cpp:357:** +1
```cpp
if (!prepare_failed && !result_status.ok() &&
^
```
**be/src/runtime/group_commit_mgr.cpp:360:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:360:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:361:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:361:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:366:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:367:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:367:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:368:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (write_wal) {
^
```
**be/src/runtime/group_commit_mgr.cpp:373:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:373:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:378:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:380:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:384:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
**be/src/runtime/group_commit_mgr.cpp:384:** +1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
</details>
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -472,4 +502,10 @@
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough() {
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:74:
```diff
- bool is_wal_disk_space_enough();
+ static bool is_wal_disk_space_enough();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869067082
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1420887919
##########
be/src/olap/wal_manager.cpp:
##########
@@ -75,11 +97,45 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ RETURN_IF_ERROR(init_wal_limit());
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_limit() {
Review Comment:
warning: method 'init_wal_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:78:
```diff
- Status init_wal_limit();
+ static Status init_wal_limit();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -155,7 +211,25 @@
Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
const std::string& label) {
std::string base_path =
- _wal_dirs.size() == 1 ? _wal_dirs[0] : _wal_dirs[rand() % _wal_dirs.size()];
+ _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::max_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_dir_to_disk_usage_map[dir1]->load() <
+ _wal_dir_to_disk_usage_map[dir2]->load();
+ });
+ std::stringstream ss;
+ ss << base_path << "/" << std::to_string(db_id) << "/" << std::to_string(table_id) << "/"
+ << std::to_string(wal_id) << "_" << label;
+ {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_lock);
+ _wal_path_map.emplace(wal_id, ss.str());
+ }
+ return Status::OK();
+}
+
+Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
Review Comment:
warning: method 'add_wal_path' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::add_wal_path(int64_t db_id, int64_t table_id, int64_t wal_id,
```
##########
be/src/olap/wal_writer.cpp:
##########
@@ -26,21 +26,15 @@
#include "io/fs/local_file_system.h"
#include "io/fs/path.h"
#include "olap/storage_engine.h"
+#include "olap/wal_manager.h"
#include "util/crc32c.h"
namespace doris {
const char* k_wal_magic = "WAL1";
const uint32_t k_wal_magic_length = 4;
-WalWriter::WalWriter(const std::string& file_name,
- const std::shared_ptr<std::atomic_size_t>& all_wal_disk_bytes,
- const std::shared_ptr<std::condition_variable>& cv)
- : cv(cv),
- _file_name(file_name),
- _disk_bytes(0),
- _all_wal_disk_bytes(all_wal_disk_bytes),
- _is_first_append_blocks(true) {}
+WalWriter::WalWriter(const std::string& file_name) : _file_name(file_name) {}
Review Comment:
warning: pass by value and use std::move [modernize-pass-by-value]
be/src/olap/wal_writer.cpp:22:
```diff
+ #include <utility>
```
```suggestion
WalWriter::WalWriter(std::string file_name) : _file_name(std::move(file_name)std::string Writer(const std::string& file_name) : _file_name(file_name) {}
```
##########
be/src/runtime/group_commit_mgr.h:
##########
@@ -20,9 +20,13 @@
#include <gen_cpp/PaloInternalService_types.h>
Review Comment:
warning: 'gen_cpp/PaloInternalService_types.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/PaloInternalService_types.h>
^
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -420,9 +426,10 @@ void GroupCommitMgr::stop() {
LOG(INFO) << "GroupCommitMgr is stopped";
}
-Status GroupCommitMgr::get_first_block_load_queue(
- int64_t db_id, int64_t table_id, std::shared_ptr<vectorized::FutureBlock> block,
- std::shared_ptr<LoadBlockQueue>& load_block_queue) {
+Status GroupCommitMgr::get_first_block_load_queue(int64_t db_id, int64_t table_id,
Review Comment:
warning: method 'get_first_block_load_queue' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status GroupCommitMgr::get_first_block_load_queue(int64_t db_id, int64_t table_id,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1407950529
##########
be/src/olap/wal_manager.cpp:
##########
@@ -75,11 +76,37 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ // Get the root path available space.
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info("./", &disk_capacity_bytes,
+ &available_bytes));
+ RETURN_IF_ERROR(init_wal_limit(available_bytes));
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_limit(size_t available_bytes) {
Review Comment:
warning: method 'init_wal_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:75:
```diff
- Status init_wal_limit(size_t available_bytes);
+ static Status init_wal_limit(size_t available_bytes);
```
##########
be/src/olap/wal_manager.h:
##########
@@ -15,9 +15,12 @@
// specific language governing permissions and limitations
// under the License.
+#pragma once
+
#include <gen_cpp/PaloInternalService_types.h>
Review Comment:
warning: 'gen_cpp/PaloInternalService_types.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/PaloInternalService_types.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831958464
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1834408827
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit fd6016c0dd628f8ee5fe136389974c1f7fb263dd, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4894 4644 4661 4644
q2 356 156 159 156
q3 1502 1305 1288 1288
q4 1152 966 920 920
q5 3260 3257 3249 3249
q6 253 130 134 130
q7 1036 523 562 523
q8 2278 2266 2229 2229
q9 6941 7070 6951 6951
q10 3298 3371 3361 3361
q11 338 213 218 213
q12 353 219 219 219
q13 4681 3857 3911 3857
q14 255 216 222 216
q15 583 544 527 527
q16 454 429 394 394
q17 1034 643 618 618
q18 8490 7832 7443 7443
q19 1566 1528 1561 1528
q20 568 383 334 334
q21 3457 2996 2970 2970
q22 368 292 313 292
Total cold run time: 47117 ms
Total hot run time: 42062 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4624 4596 4615 4596
q2 320 214 252 214
q3 3776 3751 3735 3735
q4 2547 2511 2523 2511
q5 6202 6205 6199 6199
q6 247 126 126 126
q7 2613 2011 1981 1981
q8 3759 3750 3742 3742
q9 9398 9462 9389 9389
q10 4040 4164 4144 4144
q11 642 494 489 489
q12 813 628 621 621
q13 4385 3643 3624 3624
q14 280 237 242 237
q15 581 531 531 531
q16 538 487 524 487
q17 2140 2078 2097 2078
q18 9725 9010 9034 9010
q19 1824 1754 1781 1754
q20 2318 1993 1962 1962
q21 7348 6865 6892 6865
q22 716 577 545 545
Total cold run time: 68836 ms
Total hot run time: 64840 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1841309303
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858740712
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 841af96a6e50296bfc5492d549c334879cbcb6a7, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4703 4498 4488 4488
q2 363 153 139 139
q3 1464 1282 1274 1274
q4 1119 903 872 872
q5 3163 3144 3165 3144
q6 248 129 125 125
q7 1007 492 480 480
q8 2188 2192 2198 2192
q9 6672 6654 6634 6634
q10 3207 3257 3268 3257
q11 324 191 191 191
q12 353 207 206 206
q13 4533 3820 3789 3789
q14 247 218 217 217
q15 579 528 523 523
q16 435 381 388 381
q17 1017 661 599 599
q18 6971 6878 6617 6617
q19 1544 1421 1424 1421
q20 557 323 291 291
q21 3060 2681 2646 2646
q22 342 275 281 275
Total cold run time: 44096 ms
Total hot run time: 39761 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4404 4423 4367 4367
q2 267 165 172 165
q3 3509 3471 3470 3470
q4 2378 2367 2359 2359
q5 5689 5680 5688 5680
q6 237 122 122 122
q7 2345 1879 1826 1826
q8 3525 3522 3524 3522
q9 9024 9011 8982 8982
q10 3891 3986 3985 3985
q11 504 393 394 393
q12 758 592 593 592
q13 4281 3568 3512 3512
q14 302 257 267 257
q15 573 519 530 519
q16 510 449 466 449
q17 1862 1849 1833 1833
q18 8520 8153 8118 8118
q19 1760 1735 1746 1735
q20 2248 1941 1941 1941
q21 6477 6150 6124 6124
q22 506 412 418 412
Total cold run time: 63570 ms
Total hot run time: 60363 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1428745164
##########
be/src/olap/wal_manager.cpp:
##########
@@ -75,11 +99,49 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ RETURN_IF_ERROR(init_wal_limit());
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_limit() {
Review Comment:
warning: method 'init_wal_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:77:
```diff
- Status init_wal_limit();
+ static Status init_wal_limit();
```
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -218,5 +244,16 @@ Status GroupCommitBlockSink::_add_blocks() {
return Status::OK();
}
+const string& GroupCommitBlockSink::_get_max_disk_usage_dir(
Review Comment:
warning: method '_get_max_disk_usage_dir' can be made static [readability-convert-member-functions-to-static]
```suggestion
static const string& GroupCommitBlockSink::_get_max_disk_usage_dir(
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -166,6 +227,17 @@
return Status::OK();
}
+const string& WalManager::_get_min_disk_usage_wal_dir() {
Review Comment:
warning: method '_get_min_disk_usage_wal_dir' can be made static [readability-convert-member-functions-to-static]
```suggestion
static const string& WalManager::_get_min_disk_usage_wal_dir() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1428870556
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -245,31 +245,32 @@ Status GroupCommitTable::_create_group_commit_load(
{
load_block_queue = std::make_shared<LoadBlockQueue>(
instance_id, label, txn_id, schema_version, _all_block_queues_bytes,
- result.wait_internal_group_commit_finish, result.group_commit_interval_ms);
+ result.wait_internal_group_commit_finish, result.group_commit_interval_ms, wal_dir);
std::unique_lock l(_lock);
_load_block_queues.emplace(instance_id, load_block_queue);
_need_plan_fragment = false;
_exec_env->wal_mgr()->add_wal_status_queue(_table_id, txn_id,
WalManager::WAL_STATUS::PREPARE);
//create wal
- RETURN_IF_ERROR(
- load_block_queue->create_wal(_db_id, _table_id, txn_id, label, _exec_env->wal_mgr(),
- params.desc_tbl.slotDescriptors, be_exe_version));
+ RETURN_IF_ERROR(load_block_queue->create_wal(
+ _db_id, _table_id, txn_id, label, _exec_env->wal_mgr(),
+ params.desc_tbl.slotDescriptors, be_exe_version, wal_dir));
_cv.notify_all();
}
- st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params,
- pipeline_params);
+ st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params, pipeline_params,
+ !wal_dir.empty());
if (!st.ok()) {
static_cast<void>(_finish_group_commit_load(_db_id, _table_id, label, txn_id, instance_id,
- st, true, nullptr));
+ st, true, nullptr, !wal_dir.empty()));
}
return st;
}
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
Review Comment:
warning: function '_finish_group_commit_load' has cognitive complexity of 59 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/group_commit_mgr.cpp:275:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:282:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (state) {
^
```
**be/src/runtime/group_commit_mgr.cpp:289:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:294:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:305:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:315:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (it != _load_block_queues.end()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:318:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:318:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:319:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:319:** +1
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:330:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!st.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:336:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:338:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:338:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:339:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:339:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:348:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!prepare_failed && !result_status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:348:** +1
```cpp
if (!prepare_failed && !result_status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:350:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:350:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:351:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:351:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:356:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:357:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:357:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:358:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (write_wal) {
^
```
**be/src/runtime/group_commit_mgr.cpp:363:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:363:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:368:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:370:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:374:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
**be/src/runtime/group_commit_mgr.cpp:374:** +1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1435133953
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
+ std::vector<std::string> tmp_dirs;
+ if (_wal_dirs.empty()) {
+ // default case.
+ for (const StorePath& path : ExecEnv::GetInstance()->store_paths()) {
+ tmp_dirs.emplace_back(path.path + "/wal");
+ }
+ } else {
+ // user config must be absolute path.
+ for (const std::string& wal_dir : _wal_dirs) {
+ if (std::filesystem::path(wal_dir).is_absolute()) {
+ tmp_dirs.emplace_back(wal_dir);
+ } else {
+ return Status::InternalError(
+ "BE config group_commit_replay_wal_dir has to be absolute path!");
+ }
+ }
+ }
+ _wal_dirs = tmp_dirs;
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs() {
Review Comment:
warning: method '_init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:101:
```diff
- Status _init_wal_dirs();
+ static Status _init_wal_dirs();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -409,4 +472,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:94:
```diff
- bool is_wal_disk_space_enough();
+ static bool is_wal_disk_space_enough();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -409,4 +472,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return true;
+ }
+ }
+ return false;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
Review Comment:
warning: method 'get_max_available_size' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:96:
```diff
- size_t get_max_available_size();
+ static size_t get_max_available_size();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -344,31 +418,20 @@
}
}
-Status WalManager::delete_wal(int64_t wal_id) {
+Status WalManager::delete_wal(int64_t wal_id, size_t block_queue_pre_allocated) {
Review Comment:
warning: method 'delete_wal' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:67:
```diff
- Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
+ static Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@ void WalManager::stop() {
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
Review Comment:
warning: method '_init_wal_dirs_conf' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:100:
```diff
- Status _init_wal_dirs_conf();
+ static Status _init_wal_dirs_conf();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -80,9 +117,43 @@
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
- return Thread::create(
- "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
- &_replay_thread);
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_disk_info() {
Review Comment:
warning: method '_init_wal_disk_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:102:
```diff
- Status _init_wal_disk_info();
+ static Status _init_wal_disk_info();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868299849
TeamCity be ut coverage result:
Function Coverage: 36.61% (8560/23383)
Line Coverage: 28.68% (69623/242747)
Region Coverage: 27.66% (35996/130121)
Branch Coverage: 24.39% (18399/75448)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5ee0e280a4c6dbb488d8b1da7221546ee51903e7_5ee0e280a4c6dbb488d8b1da7221546ee51903e7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868216768
TeamCity be ut coverage result:
Function Coverage: 36.61% (8561/23383)
Line Coverage: 28.69% (69635/242741)
Region Coverage: 27.68% (36012/130119)
Branch Coverage: 24.39% (18406/75450)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fc0ad82d51128abca2df68d01cf2c7de8d877d40_fc0ad82d51128abca2df68d01cf2c7de8d877d40/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869909558
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1834424761
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.34 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.9 seconds inserted 10000000 Rows, about 358K ops/s
storage size: 17163546536 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1833432802
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.01 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17163888362 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831486493
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1841526821
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 913d9cfeefefb9178c3a0f8e2ce5e5571e64ca31, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4705 4414 4410 4410
q2 357 160 157 157
q3 1445 1232 1246 1232
q4 1114 930 907 907
q5 3132 3153 3194 3153
q6 250 133 130 130
q7 1009 491 485 485
q8 2205 2192 2161 2161
q9 6674 6664 6690 6664
q10 3210 3251 3278 3251
q11 313 212 215 212
q12 354 211 213 211
q13 5648 3820 3815 3815
q14 244 212 216 212
q15 558 523 521 521
q16 450 383 390 383
q17 999 592 605 592
q18 7525 7517 7036 7036
q19 1521 1424 1366 1366
q20 564 312 322 312
q21 3091 2680 2737 2680
q22 352 286 288 286
Total cold run time: 45720 ms
Total hot run time: 40176 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4351 4352 4335 4335
q2 265 165 189 165
q3 3526 3508 3517 3508
q4 2366 2356 2360 2356
q5 5757 5722 5743 5722
q6 238 121 122 121
q7 2362 1871 1871 1871
q8 3504 3519 3513 3513
q9 9063 8975 8970 8970
q10 3909 4005 4014 4005
q11 499 388 374 374
q12 759 604 583 583
q13 4320 3568 3560 3560
q14 278 244 267 244
q15 559 524 521 521
q16 506 457 461 457
q17 1859 1851 1827 1827
q18 8565 8268 8100 8100
q19 1747 1742 1767 1742
q20 2252 1966 1939 1939
q21 6503 6132 6134 6132
q22 488 422 411 411
Total cold run time: 63676 ms
Total hot run time: 60456 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858713224
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862514558
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.87 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17183699433 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862427345
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.33 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17183516893 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1434468281
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@ void WalManager::stop() {
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
Review Comment:
warning: method '_init_wal_dirs_conf' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:101:
```diff
- Status _init_wal_dirs_conf();
+ static Status _init_wal_dirs_conf();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +469,106 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return false;
+ }
+ }
+ return true;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
Review Comment:
warning: method 'get_max_available_size' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:97:
```diff
- size_t get_max_available_size();
+ static size_t get_max_available_size();
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -472,4 +484,52 @@ Status LoadBlockQueue::close_wal() {
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
```suggestion
static bool LoadBlockQueue::is_wal_disk_space_enough(
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
+ std::vector<std::string> tmp_dirs;
+ if (_wal_dirs.empty()) {
+ // default case.
+ for (const StorePath& path : ExecEnv::GetInstance()->store_paths()) {
+ tmp_dirs.emplace_back(path.path + "/wal");
+ }
+ } else {
+ // user config must be absolute path.
+ for (const std::string& wal_dir : _wal_dirs) {
+ if (std::filesystem::path(wal_dir).is_absolute()) {
+ tmp_dirs.emplace_back(wal_dir);
+ } else {
+ return Status::InternalError(
+ "BE config group_commit_replay_wal_dir has to be absolute path!");
+ }
+ }
+ }
+ _wal_dirs = tmp_dirs;
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs() {
Review Comment:
warning: method '_init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:102:
```diff
- Status _init_wal_dirs();
+ static Status _init_wal_dirs();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -80,9 +117,41 @@
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
- return Thread::create(
- "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
- &_replay_thread);
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_disk_info() {
Review Comment:
warning: method '_init_wal_disk_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:103:
```diff
- Status _init_wal_disk_info();
+ static Status _init_wal_disk_info();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +469,106 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return false;
+ }
+ }
+ return true;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->second->available()
+ : std::max_element(_wal_disk_info_map.begin(), _wal_disk_info_map.end(),
+ [](const auto& map1, const auto& map2) {
+ return map1.second->available() <
+ map2.second->available();
+ })
+ ->second->available();
+}
+
+std::shared_mutex& WalManager::lock_for_wal_disk_info() {
+ return _wal_disk_info_lock;
+}
+
+Status WalManager::update_wal_disk_info_map(std::shared_mutex& wal_disk_info_lock,
Review Comment:
warning: method 'update_wal_disk_info_map' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::update_wal_disk_info_map(std::shared_mutex& wal_disk_info_lock,
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +469,106 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:95:
```diff
- bool is_wal_disk_space_enough();
+ static bool is_wal_disk_space_enough();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +469,106 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return false;
+ }
+ }
+ return true;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->second->available()
+ : std::max_element(_wal_disk_info_map.begin(), _wal_disk_info_map.end(),
+ [](const auto& map1, const auto& map2) {
+ return map1.second->available() <
+ map2.second->available();
+ })
+ ->second->available();
+}
+
+std::shared_mutex& WalManager::lock_for_wal_disk_info() {
+ return _wal_disk_info_lock;
+}
+
+Status WalManager::update_wal_disk_info_map(std::shared_mutex& wal_disk_info_lock,
+ std::string wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ if (_wal_disk_info_map.find(wal_dir) != _wal_disk_info_map.end()) {
+ std::unique_lock l(wal_disk_info_lock);
+ if (limit != static_cast<size_t>(-1)) {
+ _wal_disk_info_map[wal_dir]->limit = limit;
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(
+ config::group_commit_wal_max_disk_limit, -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ int64_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->file_size(wal_dir, &wal_dir_size));
+ wal_disk_limit += wal_dir_size;
+ _wal_disk_info_map[wal_dir]->limit = wal_disk_limit;
+ }
+ if (used != static_cast<size_t>(-1)) {
+ _wal_disk_info_map[wal_dir]->used = used;
+ } else {
+ int64_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->file_size(wal_dir, &wal_dir_size));
+ _wal_disk_info_map[wal_dir]->used = wal_dir_size;
+ }
+ if (pre_allocated != static_cast<size_t>(-1)) {
+ _wal_disk_info_map[wal_dir]->pre_allocated = pre_allocated;
+ }
+ } else {
+ return Status::InternalError("Can not find wal dir in wal disk info map.");
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_disk_available_size(std::shared_mutex& wal_disk_info_lock,
Review Comment:
warning: method 'get_wal_disk_available_size' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::get_wal_disk_available_size(std::shared_mutex& wal_disk_info_lock,
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -472,4 +484,52 @@
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
+ const std::vector<std::shared_ptr<vectorized::Block>>& blocks, const TUniqueId& load_id) {
+ size_t blocks_size = 0;
+ for (auto block : blocks) {
+ blocks_size += block->bytes();
+ }
+ size_t content_length = 0;
+ Status st = ExecEnv::GetInstance()->group_commit_mgr()->get_load_info(load_id, &content_length);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get load info!";
+ }
+ size_t pre_allocated = blocks_size > 0 ? blocks_size : content_length;
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ std::shared_mutex& wal_disk_info_lock = wal_mgr->lock_for_wal_disk_info();
+ size_t available_bytes = 0;
+ st = wal_mgr->get_wal_disk_available_size(wal_disk_info_lock, wal_dir_to_disk_usage.first,
+ &available_bytes);
+ if (!st.ok()) {
+ LOG(WARNING) << "get wal disk available size filed!";
+ }
+ if (pre_allocated < available_bytes) {
+ std::shared_mutex& l = wal_mgr->lock_for_wal_disk_info();
+ RETURN_IF_ERROR(wal_mgr->update_wal_disk_info_map(l, wal_dir_to_disk_usage.first, -1, -1,
+ pre_allocated));
+ return true;
+ } else {
+ return false;
+ }
+}
+
+Status GroupCommitMgr::update_load_info(TUniqueId load_id, size_t content_length) {
Review Comment:
warning: method 'update_load_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:157:
```diff
- Status update_load_info(TUniqueId load_id, size_t content_length);
+ static Status update_load_info(TUniqueId load_id, size_t content_length);
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -472,4 +484,52 @@
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
+ const std::vector<std::shared_ptr<vectorized::Block>>& blocks, const TUniqueId& load_id) {
+ size_t blocks_size = 0;
+ for (auto block : blocks) {
+ blocks_size += block->bytes();
+ }
+ size_t content_length = 0;
+ Status st = ExecEnv::GetInstance()->group_commit_mgr()->get_load_info(load_id, &content_length);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get load info!";
+ }
+ size_t pre_allocated = blocks_size > 0 ? blocks_size : content_length;
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ std::shared_mutex& wal_disk_info_lock = wal_mgr->lock_for_wal_disk_info();
+ size_t available_bytes = 0;
+ st = wal_mgr->get_wal_disk_available_size(wal_disk_info_lock, wal_dir_to_disk_usage.first,
+ &available_bytes);
+ if (!st.ok()) {
+ LOG(WARNING) << "get wal disk available size filed!";
+ }
+ if (pre_allocated < available_bytes) {
+ std::shared_mutex& l = wal_mgr->lock_for_wal_disk_info();
+ RETURN_IF_ERROR(wal_mgr->update_wal_disk_info_map(l, wal_dir_to_disk_usage.first, -1, -1,
+ pre_allocated));
+ return true;
+ } else {
+ return false;
+ }
+}
+
+Status GroupCommitMgr::update_load_info(TUniqueId load_id, size_t content_length) {
+ std::unique_lock l(_load_info_lock);
+ if (_load_id_to_content_length_map.find(load_id) != _load_id_to_content_length_map.end()) {
+ return Status::InternalError("txn id already exists!");
+ }
+ _load_id_to_content_length_map.insert(std::make_pair(load_id, content_length));
+ return Status::OK();
+}
+
+Status GroupCommitMgr::get_load_info(TUniqueId load_id, size_t* content_length) {
Review Comment:
warning: method 'get_load_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:158:
```diff
- Status get_load_info(TUniqueId load_id, size_t* content_length);
+ static Status get_load_info(TUniqueId load_id, size_t* content_length);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848597462
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1425036466
##########
be/src/http/action/http_stream.cpp:
##########
@@ -182,12 +183,22 @@ int HttpStreamAction::on_header(HttpRequest* req) {
if (!group_commit_mode.empty() || config::wait_internal_group_commit_finish) {
ctx->group_commit = load_size_smaller_than_wal_limit(req);
if (!ctx->group_commit) {
- LOG(WARNING) << "The data size for this http load("
- << std::stol(req->header(HttpHeaders::CONTENT_LENGTH))
- << " Bytes) exceeds the WAL (Write-Ahead Log) limit ("
- << config::wal_max_disk_size * 0.8
- << " Bytes). Please set this load to \"group commit\"=false.";
- st = Status::InternalError("Http load size too large.");
+ std::stringstream err_msg;
+ const auto& map = ExecEnv::GetInstance()->wal_mgr()->get_wal_dir_to_max_limit_map();
+ err_msg << "When enable group commit, the data size can't be too large. The data size "
+ "for this http load("
+ << (req->header(HttpHeaders::CONTENT_LENGTH).empty()
+ ? 0
+ : std::stol(req->header(HttpHeaders::CONTENT_LENGTH)))
+ << " Bytes) exceeds the WAL (Write-Ahead Log) limit ("
+ << std::max_element(map.begin(), map.end(),
Review Comment:
It's better to abstract a function to get the max limit wal dir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1854368183
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1834230062
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1833375096
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 5e2495ffe9c10558a7b7931fe63ca12177c285c8, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4922 4639 4670 4639
q2 358 179 145 145
q3 1501 1286 1298 1286
q4 1149 973 950 950
q5 3254 3239 3245 3239
q6 259 134 133 133
q7 1001 514 588 514
q8 2254 2239 2222 2222
q9 6932 6980 6972 6972
q10 3319 3359 3374 3359
q11 337 218 220 218
q12 353 216 219 216
q13 4628 3916 4538 3916
q14 248 229 219 219
q15 601 528 521 521
q16 431 371 391 371
q17 1033 663 612 612
q18 8075 7952 7691 7691
q19 1558 1570 1555 1555
q20 546 1184 317 317
q21 3431 2931 2969 2931
q22 378 302 308 302
Total cold run time: 46568 ms
Total hot run time: 42328 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4621 4594 4607 4594
q2 330 207 243 207
q3 3757 3757 3767 3757
q4 2542 2526 2527 2526
q5 6214 6184 6207 6184
q6 249 120 128 120
q7 2637 1966 2006 1966
q8 3768 3745 3796 3745
q9 9462 9405 9402 9402
q10 4057 4171 4150 4150
q11 671 520 530 520
q12 803 647 648 647
q13 4413 3619 3609 3609
q14 272 266 264 264
q15 587 529 526 526
q16 535 490 504 490
q17 2159 2101 2105 2101
q18 9643 9216 8965 8965
q19 1804 1767 1776 1767
q20 2330 1994 1968 1968
q21 7495 7027 7099 7027
q22 632 614 599 599
Total cold run time: 68981 ms
Total hot run time: 65134 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831400802
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.74 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17099258544 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1830060654
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831673982
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.55 seconds
stream load tsv: 556 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162408865 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848337210
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 6d983eb979ae52b3647fa7b0a29095a8dd4a3edf, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4702 4546 4514 4514
q2 365 176 151 151
q3 1445 1220 1252 1220
q4 1090 919 880 880
q5 3145 3105 3115 3105
q6 252 127 130 127
q7 984 480 484 480
q8 2182 2208 2142 2142
q9 6631 6675 6605 6605
q10 3218 3233 3287 3233
q11 327 214 208 208
q12 348 205 210 205
q13 4579 3822 3801 3801
q14 246 207 210 207
q15 568 532 525 525
q16 441 385 402 385
q17 1001 577 549 549
q18 7489 7053 6965 6965
q19 1512 1348 1439 1348
q20 543 327 306 306
q21 3035 2641 2649 2641
q22 351 281 288 281
Total cold run time: 44454 ms
Total hot run time: 39878 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4436 4442 4417 4417
q2 272 165 171 165
q3 3518 3525 3506 3506
q4 2370 2353 2366 2353
q5 5721 5697 5711 5697
q6 235 121 125 121
q7 2385 1888 1864 1864
q8 3483 3488 3498 3488
q9 9010 9010 8966 8966
q10 3921 3993 3988 3988
q11 494 395 400 395
q12 769 618 598 598
q13 4289 3550 3606 3550
q14 288 262 260 260
q15 571 523 522 522
q16 493 448 467 448
q17 1890 1858 1871 1858
q18 8694 8175 8177 8175
q19 1692 1715 1714 1714
q20 2250 1933 1963 1933
q21 6503 6180 6157 6157
q22 502 425 425 425
Total cold run time: 63786 ms
Total hot run time: 60600 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859487204
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 6cddf0bc248284ed728e18295dcdf5c9c33e7da2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4702 4456 4452 4452
q2 360 153 158 153
q3 1443 1269 1219 1219
q4 1118 918 877 877
q5 3147 3155 3134 3134
q6 248 128 128 128
q7 984 480 491 480
q8 2203 2173 2162 2162
q9 6692 6669 6625 6625
q10 3232 3276 3247 3247
q11 329 212 215 212
q12 357 210 211 210
q13 4544 3808 3828 3808
q14 240 213 209 209
q15 558 519 512 512
q16 440 387 392 387
q17 1004 641 583 583
q18 7247 7041 6969 6969
q19 1531 1443 1363 1363
q20 531 299 312 299
q21 3054 2764 2678 2678
q22 346 280 285 280
Total cold run time: 44310 ms
Total hot run time: 39987 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4391 4420 4360 4360
q2 270 166 174 166
q3 3538 3522 3510 3510
q4 2388 2389 2379 2379
q5 5715 5744 5736 5736
q6 241 120 123 120
q7 2364 1912 1874 1874
q8 3510 3519 3500 3500
q9 8968 8975 8904 8904
q10 3899 4017 3972 3972
q11 492 372 383 372
q12 759 604 595 595
q13 4302 3566 3536 3536
q14 280 249 260 249
q15 564 511 527 511
q16 508 471 477 471
q17 1883 1847 1836 1836
q18 8670 8314 8317 8314
q19 1748 1740 1728 1728
q20 2266 1952 1951 1951
q21 6500 6164 6147 6147
q22 495 415 415 415
Total cold run time: 63751 ms
Total hot run time: 60646 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1429944314
##########
be/src/common/config.cpp:
##########
@@ -1134,8 +1134,9 @@ DEFINE_String(default_tzfiles_path, "${DORIS_HOME}/zoneinfo");
// Max size(bytes) of group commit queues, used for mem back pressure, defult 64M.
DEFINE_Int32(group_commit_max_queue_size, "67108864");
-// Max size(bytes) of wal disk using, used for disk space back pressure, default 64M.
-DEFINE_Int32(wal_max_disk_size, "67108864");
+// Max size(bytes) or percentage(%) of wal disk usage, used for disk space back pressure, default 10% of the disk available space.
+// wal_max_disk_limit=1024 or wal_max_disk_limit=10% can be automatically identified.
+DEFINE_String(wal_max_disk_limit, "10%");
Review Comment:
bettern rename it `group_commit_wal_max_disk_limit`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1867765459
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859081639
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1426110483
##########
be/src/http/action/http_stream.cpp:
##########
@@ -169,25 +170,39 @@ int HttpStreamAction::on_header(HttpRequest* req) {
Status st = Status::OK();
std::string group_commit_mode = req->header(HTTP_GROUP_COMMIT);
- if (iequal(group_commit_mode, "off_mode")) {
- group_commit_mode = "";
- }
if (!group_commit_mode.empty() && !iequal(group_commit_mode, "sync_mode") &&
!iequal(group_commit_mode, "async_mode") && !iequal(group_commit_mode, "off_mode")) {
st = Status::InternalError("group_commit can only be [async_mode, sync_mode, off_mode]");
- if (iequal(group_commit_mode, "off_mode")) {
- group_commit_mode = "";
- }
+ } else if (group_commit_mode.empty() || iequal(group_commit_mode, "off_mode")) {
+ group_commit_mode = "off_mode";
Review Comment:
set group_commit_mode = "", otherwise it is confilct with line line 184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1428683809
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -218,5 +232,16 @@ Status GroupCommitBlockSink::_add_blocks() {
return Status::OK();
}
+const string& GroupCommitBlockSink::_get_min_disk_usage_dir(
Review Comment:
warning: method '_get_min_disk_usage_dir' can be made static [readability-convert-member-functions-to-static]
```suggestion
static const string& GroupCommitBlockSink::_get_min_disk_usage_dir(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858770671
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 292e1feb2c737ed1308cf93736a1564f09d21a7c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4717 4465 4427 4427
q2 362 155 145 145
q3 1442 1191 1219 1191
q4 1111 914 884 884
q5 3153 3173 3163 3163
q6 246 131 127 127
q7 1005 483 491 483
q8 2188 2232 2239 2232
q9 6710 6680 6700 6680
q10 3224 3258 3290 3258
q11 328 205 190 190
q12 353 206 213 206
q13 4564 3786 3807 3786
q14 235 218 213 213
q15 574 530 531 530
q16 444 385 383 383
q17 1018 648 589 589
q18 7068 6848 6847 6847
q19 1544 1465 1440 1440
q20 533 315 307 307
q21 3099 2699 2691 2691
q22 344 278 285 278
Total cold run time: 44262 ms
Total hot run time: 40050 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4404 4420 4366 4366
q2 272 162 165 162
q3 3479 3477 3486 3477
q4 2376 2347 2363 2347
q5 5690 5674 5686 5674
q6 239 123 123 123
q7 2369 1824 1824 1824
q8 3528 3529 3528 3528
q9 9077 8991 8974 8974
q10 3917 3980 3999 3980
q11 503 382 369 369
q12 779 594 581 581
q13 4289 3588 3565 3565
q14 286 258 260 258
q15 573 517 520 517
q16 503 455 452 452
q17 1882 1845 1836 1836
q18 8547 8092 8173 8092
q19 1751 1830 1762 1762
q20 2236 1948 1924 1924
q21 6514 6125 6108 6108
q22 488 419 433 419
Total cold run time: 63702 ms
Total hot run time: 60338 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1434706359
##########
be/src/olap/wal_writer.h:
##########
@@ -34,34 +34,27 @@ extern const uint32_t k_wal_magic_length;
class WalWriter {
public:
- explicit WalWriter(const std::string& file_name,
- const std::shared_ptr<std::atomic_size_t>& all_wal_disk_bytes,
- const std::shared_ptr<std::condition_variable>& cv);
+ explicit WalWriter(const std::string& file_name);
~WalWriter();
Status init();
Status finalize();
Status append_blocks(const PBlockArray& blocks);
- size_t disk_bytes() const { return _disk_bytes.load(std::memory_order_relaxed); };
Status append_header(uint32_t version, std::string col_ids);
std::string file_name() { return _file_name; };
public:
static const int64_t LENGTH_SIZE = 8;
static const int64_t CHECKSUM_SIZE = 4;
- std::shared_ptr<std::condition_variable> cv;
static const int64_t VERSION_SIZE = 4;
private:
static constexpr size_t MAX_WAL_WRITE_WAIT_TIME = 1000;
std::string _file_name;
io::FileWriterPtr _file_writer;
- std::atomic_size_t _disk_bytes;
- std::shared_ptr<std::atomic_size_t> _all_wal_disk_bytes;
std::mutex _mutex;
Review Comment:
is this unsed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1867877751
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 433cec4ac3c3b4239d6ab094742e7604d52d4456, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4738 4436 4430 4430
q2 372 140 161 140
q3 1466 1258 1228 1228
q4 1107 920 935 920
q5 3153 3179 3173 3173
q6 248 134 129 129
q7 1011 485 493 485
q8 2174 2250 2232 2232
q9 6700 6675 6666 6666
q10 3239 3267 3291 3267
q11 309 198 181 181
q12 357 209 212 209
q13 4559 3825 3787 3787
q14 245 215 217 215
q15 578 527 523 523
q16 438 387 382 382
q17 1013 572 554 554
q18 7086 6753 6792 6753
q19 1557 1424 1449 1424
q20 556 311 290 290
q21 3098 2665 2697 2665
q22 355 277 287 277
Total cold run time: 44359 ms
Total hot run time: 39930 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4368 4342 4407 4342
q2 269 176 170 170
q3 3528 3511 3501 3501
q4 2401 2367 2380 2367
q5 5729 5705 5707 5705
q6 242 127 124 124
q7 2359 1875 1862 1862
q8 3540 3526 3526 3526
q9 9028 9041 8999 8999
q10 3945 4041 4012 4012
q11 489 355 369 355
q12 761 592 597 592
q13 4282 3583 3575 3575
q14 291 255 250 250
q15 570 532 528 528
q16 486 460 483 460
q17 1897 1853 1847 1847
q18 8532 8196 8185 8185
q19 1774 1764 1750 1750
q20 2233 1950 1946 1946
q21 6492 6152 6151 6151
q22 495 450 453 450
Total cold run time: 63711 ms
Total hot run time: 60697 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1857367004
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit c9a693b997ad83a821b244ce4210624ff34a087d, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4734 4457 4541 4457
q2 373 142 158 142
q3 1470 1281 1191 1191
q4 1106 893 935 893
q5 3167 3219 3180 3180
q6 250 127 131 127
q7 997 499 475 475
q8 2181 2172 2182 2172
q9 6684 6677 6641 6641
q10 3229 3275 3266 3266
q11 328 199 210 199
q12 353 207 210 207
q13 4545 3798 3804 3798
q14 250 215 213 213
q15 567 526 530 526
q16 443 388 384 384
q17 1013 676 589 589
q18 7129 6756 6783 6756
q19 1547 1423 1363 1363
q20 553 309 293 293
q21 3059 2647 2629 2629
q22 351 278 288 278
Total cold run time: 44329 ms
Total hot run time: 39779 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4418 4411 4426 4411
q2 263 162 171 162
q3 3515 3506 3498 3498
q4 2387 2374 2381 2374
q5 5723 5746 5721 5721
q6 237 121 123 121
q7 2374 1843 1836 1836
q8 3511 3537 3522 3522
q9 9211 8970 9030 8970
q10 3929 3985 4027 3985
q11 499 378 395 378
q12 755 582 586 582
q13 4267 3551 3567 3551
q14 288 262 250 250
q15 576 524 526 524
q16 538 488 458 458
q17 1868 1872 1837 1837
q18 8476 8195 8132 8132
q19 1703 1724 1752 1724
q20 2265 1975 1925 1925
q21 6500 6144 6129 6129
q22 503 410 433 410
Total cold run time: 63806 ms
Total hot run time: 60500 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1431258775
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +123,93 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:126:** 87 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858890666
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit f76e0e24f92bccd9de4d6b46808ad53805ae5014, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4738 4462 4510 4462
q2 361 153 158 153
q3 1461 1273 1219 1219
q4 1106 880 881 880
q5 3141 3124 3128 3124
q6 247 128 126 126
q7 983 486 479 479
q8 2215 2196 2185 2185
q9 6670 6656 6668 6656
q10 3202 3285 3236 3236
q11 320 203 207 203
q12 348 201 210 201
q13 4553 3810 3797 3797
q14 243 219 214 214
q15 561 533 532 532
q16 437 378 391 378
q17 1017 587 511 511
q18 7158 6974 7082 6974
q19 1535 1395 1434 1395
q20 552 327 301 301
q21 3044 2671 2681 2671
q22 355 277 280 277
Total cold run time: 44247 ms
Total hot run time: 39974 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4422 4429 4412 4412
q2 266 164 171 164
q3 3526 3524 3510 3510
q4 2391 2388 2381 2381
q5 5735 5734 5732 5732
q6 241 124 124 124
q7 2388 1882 1867 1867
q8 3516 3517 3510 3510
q9 9004 8920 8974 8920
q10 3935 3968 3979 3968
q11 504 387 375 375
q12 768 603 601 601
q13 4291 3557 3576 3557
q14 279 266 258 258
q15 578 529 526 526
q16 496 448 479 448
q17 1881 1839 1847 1839
q18 8716 8233 8211 8211
q19 1723 1770 1747 1747
q20 2273 1956 1952 1952
q21 6534 6186 6120 6120
q22 513 417 421 417
Total cold run time: 63980 ms
Total hot run time: 60639 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858877851
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1430949149
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +474,36 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:84:
```diff
- bool is_wal_disk_space_enough();
+ static bool is_wal_disk_space_enough();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +474,36 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(wal_back_pressure_lock);
+ for (const auto& pair : _wal_dir_to_disk_usage_map) {
+ if (_wal_dir_to_max_limit_map.contains(pair.first) &&
+ *pair.second >= _wal_dir_to_max_limit_map[pair.first] * 0.8) {
+ return false;
+ }
+ }
+ return true;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock<std::shared_mutex> l(wal_back_pressure_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_dir_to_disk_usage_map[dir1]->load() <
+ _wal_dir_to_disk_usage_map[dir2]->load();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
Review Comment:
warning: method 'get_random_wal_dir' can be made static [readability-convert-member-functions-to-static]
```suggestion
static const string& WalManager::get_random_wal_dir() {
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -257,19 +261,20 @@ Status GroupCommitTable::_create_group_commit_load(
params.desc_tbl.slotDescriptors, be_exe_version));
_cv.notify_all();
}
- st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params,
- pipeline_params);
+ st = _exec_plan_fragment(_db_id, _table_id, label, txn_id, is_pipeline, params, pipeline_params,
+ write_wal);
if (!st.ok()) {
static_cast<void>(_finish_group_commit_load(_db_id, _table_id, label, txn_id, instance_id,
- st, true, nullptr));
+ st, true, nullptr, write_wal));
}
return st;
}
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
Review Comment:
warning: function '_finish_group_commit_load' has cognitive complexity of 74 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status GroupCommitTable::_finish_group_commit_load(int64_t db_id, int64_t table_id,
^
```
<details>
<summary>Additional context</summary>
**be/src/runtime/group_commit_mgr.cpp:279:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:286:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (state) {
^
```
**be/src/runtime/group_commit_mgr.cpp:293:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:298:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:309:** nesting level increased to 2
```cpp
[&request, &result](FrontendServiceConnection& client) {
^
```
**be/src/runtime/group_commit_mgr.cpp:319:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (it != _load_block_queues.end()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:322:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:322:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(load_block_queue->close_wal());
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:323:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:323:** +1
```cpp
if (prepare_failed || !status.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:334:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!st.ok()) {
^
```
**be/src/runtime/group_commit_mgr.cpp:340:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (!prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:342:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:342:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:343:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:343:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:348:** +1, nesting level increased to 2
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:349:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:349:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:350:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:350:** +4, including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:355:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (!prepare_failed && !result_status.ok() &&
^
```
**be/src/runtime/group_commit_mgr.cpp:355:** +1
```cpp
if (!prepare_failed && !result_status.ok() &&
^
```
**be/src/runtime/group_commit_mgr.cpp:358:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:358:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->get_wal_path(txn_id, wal_path));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:359:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:359:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->add_recover_wal(std::to_string(db_id),
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:364:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:365:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:365:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->delete_wal(txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:366:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
if (write_wal) {
^
```
**be/src/runtime/group_commit_mgr.cpp:371:** +2, including nesting penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:532:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do { \
^
```
**be/src/runtime/group_commit_mgr.cpp:371:** +3, including nesting penalty of 2, nesting level increased to 3
```cpp
RETURN_IF_ERROR(_exec_env->wal_mgr()->erase_wal_status_queue(table_id, txn_id));
^
```
**be/src/common/status.h:534:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/runtime/group_commit_mgr.cpp:376:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (prepare_failed) {
^
```
**be/src/runtime/group_commit_mgr.cpp:378:** +1, nesting level increased to 1
```cpp
} else {
^
```
**be/src/runtime/group_commit_mgr.cpp:382:** +1, including nesting penalty of 0, nesting level increased to 1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
**be/src/runtime/group_commit_mgr.cpp:382:** +1
```cpp
if (state && !(state->get_error_log_file_path().empty())) {
^
```
</details>
##########
be/src/olap/wal_manager.cpp:
##########
@@ -80,11 +87,72 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ RETURN_IF_ERROR(init_wal_limit());
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_dirs() {
Review Comment:
warning: method 'init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:79:
```diff
- Status init_wal_dirs();
+ static Status init_wal_dirs();
```
##########
be/src/runtime/group_commit_mgr.h:
##########
@@ -46,7 +52,15 @@ class LoadBlockQueue {
_start_time(std::chrono::steady_clock::now()),
_all_block_queues_bytes(all_block_queues_bytes),
_group_commit_interval_ms(group_commit_interval_ms) {
- _single_block_queue_bytes = std::make_shared<std::atomic_size_t>(0);
+ single_block_queue_bytes = std::make_shared<std::atomic_size_t>(0);
+ if (write_wal) {
+ auto wal_mgr = ExecEnv::GetInstance()->wal_mgr();
Review Comment:
warning: 'auto wal_mgr' can be declared as 'auto *wal_mgr' [readability-qualified-auto]
```suggestion
auto *wal_mgr = ExecEnv::GetInstance()->wal_mgr();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +474,36 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(wal_back_pressure_lock);
+ for (const auto& pair : _wal_dir_to_disk_usage_map) {
+ if (_wal_dir_to_max_limit_map.contains(pair.first) &&
+ *pair.second >= _wal_dir_to_max_limit_map[pair.first] * 0.8) {
+ return false;
+ }
+ }
+ return true;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
Review Comment:
warning: method 'get_min_disk_usage_wal_dir' can be made static [readability-convert-member-functions-to-static]
```suggestion
static const string& WalManager::get_min_disk_usage_wal_dir() {
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -472,4 +486,22 @@
}
return Status::OK();
}
+
+Status LoadBlockQueue::wal_back_pressure() {
Review Comment:
warning: method 'wal_back_pressure' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:74:
```diff
- Status wal_back_pressure();
+ static Status wal_back_pressure();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1833697969
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 0f2c7a7eeef3cbb7af014d7da32d91c9cd28ef94, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4895 4639 4688 4639
q2 348 168 158 158
q3 1516 1295 1267 1267
q4 1151 941 908 908
q5 3221 3231 3231 3231
q6 246 129 129 129
q7 986 520 550 520
q8 2290 2244 2237 2237
q9 7275 7011 7614 7011
q10 3302 3379 3381 3379
q11 350 210 210 210
q12 353 212 212 212
q13 4626 3861 3864 3861
q14 249 225 219 219
q15 593 548 537 537
q16 448 389 404 389
q17 1035 666 593 593
q18 8269 7706 7545 7545
q19 1567 1565 1538 1538
q20 526 354 319 319
q21 3438 2990 2977 2977
q22 373 297 304 297
Total cold run time: 47057 ms
Total hot run time: 42176 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4582 4604 4584 4584
q2 313 198 246 198
q3 3761 3752 3738 3738
q4 2540 2533 2519 2519
q5 6206 6211 6207 6207
q6 243 122 125 122
q7 2590 1951 1958 1951
q8 3763 3757 3706 3706
q9 9490 9483 9468 9468
q10 4060 4168 4152 4152
q11 638 513 519 513
q12 819 609 634 609
q13 4402 3643 3692 3643
q14 275 245 248 245
q15 589 535 530 530
q16 528 488 471 471
q17 2114 2082 2120 2082
q18 9612 9020 9058 9020
q19 1804 1790 1792 1790
q20 2313 2009 1986 1986
q21 7330 6901 6925 6901
q22 679 577 565 565
Total cold run time: 68651 ms
Total hot run time: 65000 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1436067489
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
Review Comment:
warning: method 'get_limit' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::get_limit() const {
```
be/src/olap/wal_dirs_info.h:41:
```diff
- size_t get_limit();
+ size_t get_limit() const;
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
Review Comment:
warning: method 'available' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::available() const {
```
be/src/olap/wal_dirs_info.h:47:
```diff
- size_t available();
+ size_t available() const;
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_disk_info(size_t limit, size_t used, size_t pre_allocated,
+ bool is_add_pre_allocated) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ if (pre_allocated != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_disk_info(std::string wal_dir, size_t limit, size_t used,
Review Comment:
warning: method 'update_wal_disk_info' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_wal_disk_info(std::string wal_dir, size_t limit, size_t used,
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -344,31 +418,20 @@
}
}
-Status WalManager::delete_wal(int64_t wal_id) {
+Status WalManager::delete_wal(int64_t wal_id, size_t block_queue_pre_allocated) {
Review Comment:
warning: method 'delete_wal' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:56:
```diff
- Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
+ static Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status set_used(size_t used);
^
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +75,37 @@ void WalManager::stop() {
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_dirs_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
Review Comment:
warning: method '_init_wal_dirs_conf' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:86:
```diff
- Status _init_wal_dirs_conf();
+ static Status _init_wal_dirs_conf();
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_disk_info(size_t limit, size_t used, size_t pre_allocated,
+ bool is_add_pre_allocated) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ if (pre_allocated != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_disk_info(std::string wal_dir, size_t limit, size_t used,
+ size_t pre_allocated, bool is_add_pre_allocated) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_disk_info(limit, used, pre_allocated,
+ is_add_pre_allocated);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::get_wal_disk_available_size(const std::string& wal_dir,
Review Comment:
warning: method 'get_wal_disk_available_size' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::get_wal_disk_available_size(const std::string& wal_dir,
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status set_limit(size_t limit);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
^
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_disk_info(size_t limit, size_t used, size_t pre_allocated,
Review Comment:
warning: method 'update_wal_disk_info' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirInfo::update_wal_disk_info(size_t limit, size_t used, size_t pre_allocated,
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
Review Comment:
warning: method 'get_pre_allocated' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::get_pre_allocated() const { const
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_disk_info(size_t limit = -1, size_t used = -1, size_t pre_allocated = -1,
+ bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_disk_info(std::string wal_dir, size_t limit = -1, size_t used = -1,
+ size_t pre_allocated = -1, bool is_add_pre_allocated = true);
+ Status get_wal_disk_available_size(const std::string& wal_dir, size_t* available_bytes);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status get_wal_disk_available_size(const std::string& wal_dir, size_t* available_bytes);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_disk_info(size_t limit = -1, size_t used = -1, size_t pre_allocated = -1,
+ bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
^
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,179 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
Review Comment:
warning: method 'get_used' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::get_used() const {
```
be/src/olap/wal_dirs_info.h:42:
```diff
- size_t get_used();
+ size_t get_used() const;
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +75,37 @@
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_dirs_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
+ std::vector<std::string> tmp_dirs;
+ if (_wal_dirs.empty()) {
+ // default case.
+ for (const StorePath& path : ExecEnv::GetInstance()->store_paths()) {
+ tmp_dirs.emplace_back(path.path + "/wal");
+ }
+ } else {
+ // user config must be absolute path.
+ for (const std::string& wal_dir : _wal_dirs) {
+ if (std::filesystem::path(wal_dir).is_absolute()) {
+ tmp_dirs.emplace_back(wal_dir);
+ } else {
+ return Status::InternalError(
+ "BE config group_commit_replay_wal_dir has to be absolute path!");
+ }
+ }
+ }
+ _wal_dirs = tmp_dirs;
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs() {
Review Comment:
warning: method '_init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:87:
```diff
- Status _init_wal_dirs();
+ static Status _init_wal_dirs();
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_disk_info(size_t limit = -1, size_t used = -1, size_t pre_allocated = -1,
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_disk_info(size_t limit = -1, size_t used = -1, size_t pre_allocated = -1,
^
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -475,4 +484,67 @@ Status LoadBlockQueue::close_wal() {
}
return Status::OK();
}
+
+bool LoadBlockQueue::has_enough_wal_disk_space(
Review Comment:
warning: method 'has_enough_wal_disk_space' can be made static [readability-convert-member-functions-to-static]
```suggestion
static bool LoadBlockQueue::has_enough_wal_disk_space(
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,78 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_disk_info(size_t limit = -1, size_t used = -1, size_t pre_allocated = -1,
+ bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_disk_info(std::string wal_dir, size_t limit = -1, size_t used = -1,
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_disk_info(std::string wal_dir, size_t limit = -1, size_t used = -1,
^
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -80,9 +120,39 @@
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
- return Thread::create(
- "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
- &_replay_thread);
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs_info() {
Review Comment:
warning: method '_init_wal_dirs_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:88:
```diff
- Status _init_wal_dirs_info();
+ static Status _init_wal_dirs_info();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868299348
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 5ee0e280a4c6dbb488d8b1da7221546ee51903e7, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4687 4453 4443 4443
q2 387 138 157 138
q3 1458 1278 1222 1222
q4 1105 884 884 884
q5 3178 3208 3202 3202
q6 252 130 136 130
q7 1009 486 496 486
q8 2194 2235 2215 2215
q9 6700 6695 6657 6657
q10 3205 3282 3284 3282
q11 306 192 185 185
q12 358 204 212 204
q13 4570 3799 3779 3779
q14 238 213 223 213
q15 574 524 514 514
q16 442 389 386 386
q17 1017 696 581 581
q18 7046 6748 6978 6748
q19 1535 1420 1479 1420
q20 531 312 310 310
q21 3101 2670 2644 2644
q22 345 282 281 281
Total cold run time: 44238 ms
Total hot run time: 39924 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4360 4391 4386 4386
q2 274 165 172 165
q3 3529 3498 3504 3498
q4 2408 2375 2380 2375
q5 5731 5698 5721 5698
q6 246 123 124 123
q7 2372 1836 1826 1826
q8 3533 3545 3543 3543
q9 9081 9006 9029 9006
q10 3920 3986 3995 3986
q11 488 368 371 368
q12 772 592 611 592
q13 4279 3541 3535 3535
q14 290 252 254 252
q15 567 522 520 520
q16 499 478 467 467
q17 1909 1893 1877 1877
q18 8570 8221 8140 8140
q19 1761 1769 1770 1769
q20 2250 1939 1943 1939
q21 6518 6204 6151 6151
q22 494 431 416 416
Total cold run time: 63851 ms
Total hot run time: 60632 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848413104
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 37b3ae57fc33b9d06c2f025a792c0f103c051bd5, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4648 4470 4535 4470
q2 364 177 137 137
q3 1453 1245 1202 1202
q4 1096 899 905 899
q5 3144 3132 3145 3132
q6 250 127 126 126
q7 1003 485 482 482
q8 2202 2244 2157 2157
q9 6621 6627 6629 6627
q10 3225 3245 3223 3223
q11 322 197 199 197
q12 348 206 215 206
q13 4551 3831 3794 3794
q14 237 216 240 216
q15 580 544 545 544
q16 456 385 398 385
q17 988 584 582 582
q18 7884 8679 8942 8679
q19 1513 1332 1407 1332
q20 550 453 310 310
q21 3038 2580 2720 2580
q22 346 280 289 280
Total cold run time: 44819 ms
Total hot run time: 41560 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4405 4383 4360 4360
q2 265 168 169 168
q3 3520 3501 3503 3501
q4 2384 2362 2364 2362
q5 5720 5695 5698 5695
q6 238 122 121 121
q7 2399 1870 1861 1861
q8 3478 3480 3478 3478
q9 9007 8950 8947 8947
q10 3913 3976 3991 3976
q11 505 395 392 392
q12 767 597 603 597
q13 4303 3582 3545 3545
q14 287 266 263 263
q15 574 521 528 521
q16 529 503 499 499
q17 1826 1846 1844 1844
q18 8677 8100 8431 8100
q19 1712 1723 1714 1714
q20 2248 1947 1932 1932
q21 6484 6146 6115 6115
q22 494 420 445 420
Total cold run time: 63735 ms
Total hot run time: 60411 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1854547213
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.53 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17222307891 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1852632465
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.12 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17222819758 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1852612704
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 6b84e1a88b9cda5e213cf3f83a85fbb812612706, data reload: false
run tpch-sf100 query with default conf and session variables
q1 51608 31824 31838 31824
q2 380 139 142 139
q3 1555 1258 1183 1183
q4 1154 992 873 873
q5 3299 3240 3245 3240
q6 285 157 150 150
q7 1017 534 527 527
q8 2178 2268 2174 2174
q9 7078 7032 6950 6950
q10 3265 3398 3404 3398
q11 340 219 209 209
q12 360 207 203 203
q13 4595 3972 3928 3928
q14 373 318 317 317
q15 821 1304 1342 1304
q16 461 404 384 384
q17 1025 628 567 567
q18 7488 7319 7302 7302
q19 1535 1333 1467 1333
q20 955 381 337 337
q21 3093 2616 2754 2616
q22 349 271 280 271
Total cold run time: 93214 ms
Total hot run time: 69229 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30822 31833 31809 31809
q2 278 169 178 169
q3 3685 3666 3620 3620
q4 2401 2393 2386 2386
q5 5784 5789 5796 5789
q6 244 135 132 132
q7 3048 3401 3372 3372
q8 3534 3526 3530 3526
q9 9222 9114 9159 9114
q10 3967 4045 4066 4045
q11 813 899 1365 899
q12 763 589 601 589
q13 4292 3555 3541 3541
q14 509 422 445 422
q15 1134 1330 1331 1330
q16 501 490 450 450
q17 1878 1863 1831 1831
q18 8964 8369 8269 8269
q19 1792 1756 1764 1756
q20 2903 2617 2657 2617
q21 6532 6177 6141 6141
q22 524 416 412 412
Total cold run time: 93590 ms
Total hot run time: 92219 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1425111937
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -200,9 +200,31 @@ Status GroupCommitBlockSink::_add_blocks() {
load_id.__set_lo(_load_id.lo);
if (_load_block_queue == nullptr) {
if (_state->exec_env()->wal_mgr()->is_running()) {
+ std::string wal_dir;
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ {
+ std::unique_lock l(lock);
+ auto map = wal_mgr->get_wal_dir_to_disk_usage_map();
+ wal_dir = map.size() == 1
+ ? map.begin()->first
+ : std::max_element(map.begin(), map.end(),
+ [](const auto& pair1, const auto& pair2) {
+ return pair1.second->load() <
+ pair2.second->load();
+ })
+ ->first;
+ while (map.at(wal_dir)->load() >
+ wal_mgr->get_wal_dir_to_max_limit_map().at(wal_dir)) {
+ ExecEnv::GetInstance()->group_commit_mgr()->cv.wait_for(
+ l, std::chrono::milliseconds(1000));
+ }
+ }
+
RETURN_IF_ERROR(_state->exec_env()->group_commit_mgr()->get_first_block_load_queue(
_db_id, _table_id, _base_schema_version, load_id, _load_block_queue,
- _state->be_exec_version()));
+ _state->be_exec_version(), wal_dir));
+ RETURN_IF_ERROR(wal_mgr->add_wal_path(_db_id, _table_id, _load_block_queue->txn_id,
Review Comment:
can move this to `LoadBlockQueue::create_wal` to hide the implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858802900
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 057414855af5c11acc8236b5b1324e5e5ee3aa8e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4722 4441 4441 4441
q2 378 142 149 142
q3 1451 1233 1244 1233
q4 1105 917 902 902
q5 3114 3149 3155 3149
q6 247 128 126 126
q7 1011 488 475 475
q8 2177 2245 2218 2218
q9 6676 6641 6667 6641
q10 3209 3276 3262 3262
q11 317 205 210 205
q12 354 206 217 206
q13 4554 3790 3799 3790
q14 238 208 213 208
q15 567 536 520 520
q16 447 379 391 379
q17 1021 615 495 495
q18 7367 7087 7918 7087
q19 1544 1397 1418 1397
q20 511 296 297 296
q21 3051 2647 2656 2647
q22 350 291 289 289
Total cold run time: 44411 ms
Total hot run time: 40108 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4399 4409 4390 4390
q2 268 163 168 163
q3 3532 3525 3521 3521
q4 2372 2363 2359 2359
q5 5728 5723 5708 5708
q6 241 121 120 120
q7 2396 1907 1869 1869
q8 3497 3522 3500 3500
q9 9091 9015 9034 9015
q10 3910 4002 3993 3993
q11 497 392 384 384
q12 758 596 606 596
q13 4301 3527 3564 3527
q14 282 249 265 249
q15 572 523 535 523
q16 511 461 458 458
q17 1865 1842 1862 1842
q18 8680 8270 8354 8270
q19 1734 1753 1729 1729
q20 2264 1959 1931 1931
q21 6498 6159 6153 6153
q22 503 415 432 415
Total cold run time: 63899 ms
Total hot run time: 60715 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862680744
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 3ecdddb09b8f02cbb732d311a39c7a21ca72d7d7, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4722 4418 4423 4418
q2 353 161 158 158
q3 1456 1238 1238 1238
q4 1121 934 926 926
q5 3146 3172 3159 3159
q6 238 127 125 125
q7 978 488 485 485
q8 2189 2208 2189 2189
q9 6705 6642 6666 6642
q10 3232 3249 3272 3249
q11 307 192 185 185
q12 356 212 206 206
q13 4578 3793 3825 3793
q14 239 209 213 209
q15 562 528 518 518
q16 443 387 383 383
q17 990 574 589 574
q18 7237 6946 7041 6946
q19 1518 1292 1389 1292
q20 551 343 303 303
q21 3062 2667 2659 2659
q22 350 278 278 278
Total cold run time: 44333 ms
Total hot run time: 39935 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4351 4342 4327 4327
q2 271 169 170 169
q3 3535 3536 3532 3532
q4 2391 2381 2376 2376
q5 5755 5785 5750 5750
q6 237 120 119 119
q7 2397 1864 1922 1864
q8 3507 3526 3518 3518
q9 9107 8937 8998 8937
q10 3907 3996 4027 3996
q11 497 375 370 370
q12 767 601 598 598
q13 4280 3542 3563 3542
q14 278 257 264 257
q15 582 526 522 522
q16 509 476 489 476
q17 1878 1872 1851 1851
q18 8654 8259 8147 8147
q19 1724 1753 1738 1738
q20 2251 1937 1925 1925
q21 6542 6171 6187 6171
q22 515 450 428 428
Total cold run time: 63935 ms
Total hot run time: 60613 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868219982
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17184096652 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1867885412
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.15 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.8 seconds inserted 10000000 Rows, about 335K ops/s
storage size: 17184056968 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868316296
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868294892
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1830371555
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.13 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17098848859 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831487776
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1832279414
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.2 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17100160776 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1833278797
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858788824
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859008372
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 5281c7c3b2604b2d8719c6cc8f6b9651b8ce6123, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4724 4480 4495 4480
q2 371 150 162 150
q3 1447 1247 1213 1213
q4 1117 890 895 890
q5 3157 3123 3161 3123
q6 242 126 125 125
q7 975 485 483 483
q8 2208 2206 2181 2181
q9 6706 6643 6671 6643
q10 3222 3269 3279 3269
q11 333 207 199 199
q12 355 203 199 199
q13 4564 3778 3802 3778
q14 236 210 213 210
q15 560 529 527 527
q16 437 383 382 382
q17 1009 625 539 539
q18 7189 6969 7072 6969
q19 1526 1486 1368 1368
q20 540 308 297 297
q21 3181 2622 2655 2622
q22 346 275 280 275
Total cold run time: 44445 ms
Total hot run time: 39922 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4390 4399 4392 4392
q2 265 165 171 165
q3 3536 3527 3522 3522
q4 2383 2381 2366 2366
q5 5720 5748 5721 5721
q6 243 121 122 121
q7 2392 1870 1859 1859
q8 3507 3517 3515 3515
q9 8993 8961 8932 8932
q10 3910 4002 3978 3978
q11 500 393 389 389
q12 771 591 594 591
q13 4298 3556 3558 3556
q14 283 262 263 262
q15 563 527 521 521
q16 505 476 443 443
q17 1879 1844 1824 1824
q18 8724 8303 8256 8256
q19 1744 1754 1750 1750
q20 2251 1941 1933 1933
q21 6512 6145 6146 6145
q22 509 418 419 418
Total cold run time: 63878 ms
Total hot run time: 60659 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1410250684
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +124,95 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:127:** 89 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1418509959
##########
be/src/olap/wal_writer.cpp:
##########
@@ -133,4 +137,16 @@ Status WalWriter::append_header(uint32_t version, std::string col_ids) {
return Status::OK();
}
+std::string WalWriter::_get_base_path(const std::string& wal_path) {
Review Comment:
warning: method '_get_base_path' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_writer.h:57:
```diff
- std::string _get_base_path(const std::string& wal_path);
+ static std::string _get_base_path(const std::string& wal_path);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848282693
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848606316
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.35 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17218028535 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848611534
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 306da6d08336a7715cf5366fdd1262c95c263e1c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4670 4451 4471 4451
q2 384 161 160 160
q3 1458 1255 1256 1255
q4 1119 902 909 902
q5 3148 3111 3120 3111
q6 253 126 129 126
q7 979 481 476 476
q8 2199 2202 2172 2172
q9 6654 6733 6670 6670
q10 3218 3236 3279 3236
q11 332 200 198 198
q12 342 214 208 208
q13 4559 3777 3808 3777
q14 238 211 218 211
q15 561 522 524 522
q16 433 382 393 382
q17 1005 559 551 551
q18 7778 7224 7250 7224
q19 1508 1256 1456 1256
q20 544 368 303 303
q21 3009 2668 2647 2647
q22 356 285 292 285
Total cold run time: 44747 ms
Total hot run time: 40123 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4389 4357 4414 4357
q2 267 164 172 164
q3 3548 3550 3529 3529
q4 2407 2391 2382 2382
q5 5755 5744 5749 5744
q6 241 121 120 120
q7 2393 1890 1928 1890
q8 3516 3534 3524 3524
q9 9118 8948 8994 8948
q10 3893 3974 3966 3966
q11 510 385 402 385
q12 765 590 597 590
q13 4295 3549 3542 3542
q14 282 257 257 257
q15 563 518 520 518
q16 517 457 457 457
q17 1835 1851 1899 1851
q18 8694 8422 8518 8422
q19 1734 1725 1751 1725
q20 2273 1939 1952 1939
q21 6563 6167 6169 6167
q22 492 433 435 433
Total cold run time: 64050 ms
Total hot run time: 60910 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869231902
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1868209160
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869659470
TeamCity be ut coverage result:
Function Coverage: 36.61% (8583/23442)
Line Coverage: 28.66% (69762/243409)
Region Coverage: 27.66% (36064/130405)
Branch Coverage: 24.37% (18425/75620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3a2aede8b0a558f73b25313f5b0bd159d352432c_3a2aede8b0a558f73b25313f5b0bd159d352432c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869650928
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 3a2aede8b0a558f73b25313f5b0bd159d352432c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4714 4442 4448 4442
q2 377 180 166 166
q3 1478 1310 1315 1310
q4 1151 973 959 959
q5 3184 3144 3196 3144
q6 258 135 134 134
q7 1037 488 485 485
q8 2244 2308 2247 2247
q9 6717 6680 6699 6680
q10 3226 3276 3280 3276
q11 331 221 220 220
q12 359 206 205 205
q13 4578 3821 3818 3818
q14 247 212 220 212
q15 560 530 524 524
q16 441 387 381 381
q17 1038 788 656 656
q18 7076 6846 7765 6846
q19 1581 1605 1595 1595
q20 580 309 327 309
q21 3233 2802 2801 2801
q22 380 302 311 302
Total cold run time: 44790 ms
Total hot run time: 40712 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4403 4438 4382 4382
q2 277 169 167 167
q3 3514 3513 3497 3497
q4 2445 2424 2396 2396
q5 5733 5731 5714 5714
q6 247 127 128 127
q7 2367 1830 1885 1830
q8 3607 3604 3601 3601
q9 9017 8950 9021 8950
q10 3934 4016 4034 4016
q11 479 360 364 360
q12 773 612 594 594
q13 4285 3525 3564 3525
q14 282 255 259 255
q15 567 518 523 518
q16 505 458 464 458
q17 1967 1971 1940 1940
q18 8739 8145 8284 8145
q19 1822 1837 1839 1837
q20 2279 1956 1927 1927
q21 6641 6257 6240 6240
q22 558 451 467 451
Total cold run time: 64441 ms
Total hot run time: 60930 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1869656966
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.46 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17184209698 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1436133347
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
Review Comment:
warning: method 'update_wal_dir_pre_allocated' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
Review Comment:
warning: method 'update_wal_dir_limit' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
Review Comment:
warning: method 'update_wal_dir_used' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirInfo::update_wal_dir_used(size_t used) {
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
Review Comment:
warning: method 'update_all_wal_dir_limit' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_all_wal_dir_limit() {
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
^
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
Review Comment:
warning: method 'update_wal_dir_used' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ Status update_all_wal_dir_limit();
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_all_wal_dir_limit();
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ Status update_all_wal_dir_limit();
+ Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
+ Status update_all_wal_dir_used();
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_all_wal_dir_used();
^
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -409,4 +480,46 @@
return Status::OK();
}
+size_t WalManager::get_max_available_size() {
+ return _wal_dirs_info->get_max_available_size();
+}
+
+Status WalManager::update_wal_dir_limit(const std::string& wal_dir, size_t limit) {
+ return _wal_dirs_info->update_wal_dir_limit(wal_dir, limit);
+}
+
+Status WalManager::update_wal_dir_used(const std::string& wal_dir, size_t used) {
+ return _wal_dirs_info->update_wal_dir_used(wal_dir, used);
+}
+
+Status WalManager::update_wal_dir_pre_allocated(const std::string& wal_dir, size_t pre_allocated,
+ bool is_add_pre_allocated) {
+ return _wal_dirs_info->update_wal_dir_pre_allocated(wal_dir, pre_allocated,
+ is_add_pre_allocated);
+}
+
+Status WalManager::_update_wal_dir_info_thread() {
Review Comment:
warning: method '_update_wal_dir_info_thread' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:95:
```diff
- Status _update_wal_dir_info_thread();
+ static Status _update_wal_dir_info_thread();
```
##########
be/test/olap/wal_manager_test.cpp:
##########
@@ -123,4 +123,107 @@ TEST_F(WalManagerTest, recovery_normal) {
ASSERT_TRUE(!std::filesystem::exists(wal_200));
ASSERT_TRUE(!std::filesystem::exists(wal_201));
}
+
+TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
Review Comment:
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
<details>
<summary>Additional context</summary>
**be/test/olap/wal_manager_test.cpp:126:** 101 lines including whitespace and comments (threshold 80)
```cpp
TEST_F(WalManagerTest, TestDynamicWalSpaceLimt) {
^
```
</details>
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
Review Comment:
warning: method 'update_wal_dir_limit' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirInfo::update_wal_dir_limit(size_t limit) {
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_used(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
+ bool is_add_pre_allocated) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_pre_allocated(pre_allocated, is_add_pre_allocated);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::get_wal_dir_available_size(const std::string& wal_dir,
Review Comment:
warning: method 'get_wal_dir_available_size' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::get_wal_dir_available_size(const std::string& wal_dir,
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_used(size_t used = -1);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ Status update_all_wal_dir_limit();
+ Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
^
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
Review Comment:
warning: method 'update_all_wal_dir_used' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_all_wal_dir_used() {
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_used(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
Review Comment:
warning: method 'update_wal_dir_pre_allocated' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalDirsInfo::update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_limit(size_t limit = -1);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ Status update_all_wal_dir_limit();
+ Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
+ Status update_all_wal_dir_used();
+ Status update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
+ bool is_add_pre_allocated);
+ Status get_wal_dir_available_size(const std::string& wal_dir, size_t* available_bytes);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status get_wal_dir_available_size(const std::string& wal_dir, size_t* available_bytes);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
^
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
+ std::string get_available_random_wal_dir();
+ size_t get_max_available_size();
+ Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ Status update_all_wal_dir_limit();
+ Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
+ Status update_all_wal_dir_used();
+ Status update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
^
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -60,12 +70,46 @@ void WalManager::stop() {
if (_replay_thread) {
_replay_thread->join();
}
+ if (_update_wal_dirs_info_thread) {
+ _update_wal_dirs_info_thread->join();
+ }
_thread_pool->shutdown();
LOG(INFO) << "WalManager is stopped";
}
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_dirs_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
Review Comment:
warning: method '_init_wal_dirs_conf' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:90:
```diff
- Status _init_wal_dirs_conf();
+ static Status _init_wal_dirs_conf();
```
##########
be/src/olap/wal_dirs_info.h:
##########
@@ -0,0 +1,84 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/factory_creator.h"
+
+namespace doris {
+class WalDirInfo {
+ ENABLE_FACTORY_CREATOR(WalDirInfo);
+
+public:
+ WalDirInfo(std::string wal_dir, size_t limit, size_t used, size_t pre_allocated)
+ : _wal_dir(std::move(wal_dir)),
+ _limit(limit),
+ _used(used),
+ _pre_allocated(pre_allocated) {}
+ std::string get_wal_dir();
+ size_t get_limit();
+ size_t get_used();
+ size_t get_pre_allocated();
+ Status set_limit(size_t limit);
+ Status set_used(size_t used);
+ Status set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated);
+ size_t available();
+ Status update_wal_dir_limit(size_t limit = -1);
+ Status update_wal_dir_used(size_t used = -1);
+ Status update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated = true);
+
+private:
+ std::string _wal_dir;
+ size_t _limit;
+ size_t _used;
+ size_t _pre_allocated;
+ std::shared_mutex _lock;
+};
+
+class WalDirsInfo {
+ ENABLE_FACTORY_CREATOR(WalDirsInfo);
+
+public:
+ WalDirsInfo() = default;
+ ~WalDirsInfo() = default;
+ Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ Status clear();
Review Comment:
warning: unknown type name 'Status' [clang-diagnostic-error]
```cpp
Status clear();
^
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -344,31 +426,20 @@
}
}
-Status WalManager::delete_wal(int64_t wal_id) {
+Status WalManager::delete_wal(int64_t wal_id, size_t block_queue_pre_allocated) {
Review Comment:
warning: method 'delete_wal' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:58:
```diff
- Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
+ static Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -60,12 +70,46 @@
if (_replay_thread) {
_replay_thread->join();
}
+ if (_update_wal_dirs_info_thread) {
+ _update_wal_dirs_info_thread->join();
+ }
_thread_pool->shutdown();
LOG(INFO) << "WalManager is stopped";
}
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_dirs_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
+ std::vector<std::string> tmp_dirs;
+ if (_wal_dirs.empty()) {
+ // default case.
+ for (const StorePath& path : ExecEnv::GetInstance()->store_paths()) {
+ tmp_dirs.emplace_back(path.path + "/wal");
+ }
+ } else {
+ // user config must be absolute path.
+ for (const std::string& wal_dir : _wal_dirs) {
+ if (std::filesystem::path(wal_dir).is_absolute()) {
+ tmp_dirs.emplace_back(wal_dir);
+ } else {
+ return Status::InternalError(
+ "BE config group_commit_replay_wal_dir has to be absolute path!");
+ }
+ }
+ }
+ _wal_dirs = tmp_dirs;
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs() {
Review Comment:
warning: method '_init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:91:
```diff
- Status _init_wal_dirs();
+ static Status _init_wal_dirs();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1852417077
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1425088339
##########
be/src/olap/wal_writer.h:
##########
@@ -34,34 +34,28 @@ extern const uint32_t k_wal_magic_length;
class WalWriter {
public:
- explicit WalWriter(const std::string& file_name,
- const std::shared_ptr<std::atomic_size_t>& all_wal_disk_bytes,
- const std::shared_ptr<std::condition_variable>& cv);
+ explicit WalWriter(const std::string& file_name);
~WalWriter();
Status init();
Status finalize();
Status append_blocks(const PBlockArray& blocks);
- size_t disk_bytes() const { return _disk_bytes.load(std::memory_order_relaxed); };
Status append_header(uint32_t version, std::string col_ids);
std::string file_name() { return _file_name; };
+ static std::string get_base_path(const std::string& wal_path);
Review Comment:
Is this function unused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1425092838
##########
be/src/vec/sink/group_commit_block_sink.cpp:
##########
@@ -200,9 +200,31 @@ Status GroupCommitBlockSink::_add_blocks() {
load_id.__set_lo(_load_id.lo);
if (_load_block_queue == nullptr) {
if (_state->exec_env()->wal_mgr()->is_running()) {
+ std::string wal_dir;
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ {
+ std::unique_lock l(lock);
+ auto map = wal_mgr->get_wal_dir_to_disk_usage_map();
+ wal_dir = map.size() == 1
Review Comment:
Abstract the select wal_dir to a method.
Is it better to move this method into the while loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848415845
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.05 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17219747082 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848397664
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1854529431
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 6853342f804ece8259b8f57b2ba487585500d5ac, data reload: false
run tpch-sf100 query with default conf and session variables
q1 51423 31825 31847 31825
q2 373 138 145 138
q3 1518 1252 1253 1252
q4 1103 899 871 871
q5 3198 3241 3213 3213
q6 268 140 153 140
q7 1010 502 493 493
q8 2142 2210 2187 2187
q9 6887 6868 6871 6868
q10 3247 3581 3310 3310
q11 345 220 217 217
q12 350 215 209 209
q13 4583 3803 3843 3803
q14 428 327 310 310
q15 792 1318 1346 1318
q16 455 386 387 386
q17 1016 594 648 594
q18 7519 6988 8016 6988
q19 1538 1510 1388 1388
q20 631 289 331 289
q21 3069 2647 2653 2647
q22 350 279 281 279
Total cold run time: 92245 ms
Total hot run time: 68725 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30823 31847 31802 31802
q2 283 169 176 169
q3 3619 3605 3580 3580
q4 2389 2372 2363 2363
q5 5763 5754 5758 5754
q6 241 140 130 130
q7 3061 3407 3366 3366
q8 3534 3518 3526 3518
q9 9176 9138 9134 9134
q10 3991 4030 4056 4030
q11 813 927 1351 927
q12 765 596 612 596
q13 4290 3576 3588 3576
q14 517 430 454 430
q15 1118 1349 1348 1348
q16 499 440 470 440
q17 1861 1878 1853 1853
q18 8785 8168 8103 8103
q19 1757 1798 1777 1777
q20 2888 2591 2642 2591
q21 6502 6143 6147 6143
q22 503 422 420 420
Total cold run time: 93178 ms
Total hot run time: 92050 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1408849976
##########
be/src/olap/wal_manager.cpp:
##########
@@ -75,11 +77,43 @@ Status WalManager::init() {
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
+ RETURN_IF_ERROR(init_wal_limit());
return Thread::create(
"WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
&_replay_thread);
}
+Status WalManager::init_wal_limit() {
Review Comment:
warning: method 'init_wal_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:75:
```diff
- Status init_wal_limit();
+ static Status init_wal_limit();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1831272023
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1841540171
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.91 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17162364061 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1848341765
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.44 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17223352733 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1857206389
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1857569619
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862299940
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1434962037
##########
be/src/olap/wal_manager.cpp:
##########
@@ -80,9 +117,43 @@
}
RETURN_IF_ERROR(scan_wals(wal_dir));
}
- return Thread::create(
- "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
- &_replay_thread);
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_disk_info() {
Review Comment:
warning: method '_init_wal_disk_info' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:101:
```diff
- Status _init_wal_disk_info();
+ static Status _init_wal_disk_info();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
+ std::vector<std::string> tmp_dirs;
+ if (_wal_dirs.empty()) {
+ // default case.
+ for (const StorePath& path : ExecEnv::GetInstance()->store_paths()) {
+ tmp_dirs.emplace_back(path.path + "/wal");
+ }
+ } else {
+ // user config must be absolute path.
+ for (const std::string& wal_dir : _wal_dirs) {
+ if (std::filesystem::path(wal_dir).is_absolute()) {
+ tmp_dirs.emplace_back(wal_dir);
+ } else {
+ return Status::InternalError(
+ "BE config group_commit_replay_wal_dir has to be absolute path!");
+ }
+ }
+ }
+ _wal_dirs = tmp_dirs;
+ return Status::OK();
+}
+
+Status WalManager::_init_wal_dirs() {
Review Comment:
warning: method '_init_wal_dirs' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:100:
```diff
- Status _init_wal_dirs();
+ static Status _init_wal_dirs();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +473,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return true;
+ }
+ }
+ return false;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->second->available()
+ : std::max_element(_wal_disk_info_map.begin(), _wal_disk_info_map.end(),
+ [](const auto& map1, const auto& map2) {
+ return map1.second->available() <
+ map2.second->available();
+ })
+ ->second->available();
+}
+
+Status WalManager::update_wal_disk_info_map(std::string wal_dir, size_t limit, size_t used,
+ size_t pre_allocated, bool is_add_pre_allocated) {
+ if (_wal_disk_info_map.find(wal_dir) != _wal_disk_info_map.end()) {
+ std::unique_lock l(_wal_disk_info_lock);
+ if (limit != static_cast<size_t>(-1)) {
+ _wal_disk_info_map[wal_dir]->limit = limit;
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(
+ config::group_commit_wal_max_disk_limit, -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(wal_dir, &wal_dir_size));
+ _wal_disk_info_map[wal_dir]->limit = wal_disk_limit;
+ }
+ if (used != static_cast<size_t>(-1)) {
+ _wal_disk_info_map[wal_dir]->used = used;
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(wal_dir, &wal_dir_size));
+ _wal_disk_info_map[wal_dir]->used = wal_dir_size;
+ }
+ if (pre_allocated != static_cast<size_t>(-1)) {
+ if (is_add_pre_allocated) {
+ _wal_disk_info_map[wal_dir]->pre_allocated += pre_allocated;
+ } else {
+ _wal_disk_info_map[wal_dir]->pre_allocated -= pre_allocated;
+ }
+ }
+ } else {
+ return Status::InternalError("Can not find wal dir in wal disk info map.");
+ }
+ return Status::OK();
+}
+
+Status WalManager::get_wal_disk_available_size(const std::string& wal_dir,
Review Comment:
warning: method 'get_wal_disk_available_size' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::get_wal_disk_available_size(const std::string& wal_dir,
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -495,4 +505,70 @@
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
+ const std::vector<std::shared_ptr<vectorized::Block>>& blocks, const TUniqueId& load_id,
+ bool is_blocks_contain_all_load_data) {
+ size_t blocks_size = 0;
+ for (auto block : blocks) {
+ blocks_size += block->bytes();
+ }
+ size_t content_length = 0;
+ Status st = ExecEnv::GetInstance()->group_commit_mgr()->get_load_info(load_id, &content_length);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get load info!";
+ }
+ st = ExecEnv::GetInstance()->group_commit_mgr()->remove_load_info(load_id);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to remove load info!";
+ }
+ size_t pre_allocated = is_blocks_contain_all_load_data
+ ? blocks_size
+ : (blocks_size > content_length ? blocks_size : content_length);
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ size_t available_bytes = 0;
+ {
+ st = wal_mgr->get_wal_disk_available_size(wal_base_path, &available_bytes);
+ if (!st.ok()) {
+ LOG(WARNING) << "get wal disk available size filed!";
+ }
+ }
+ if (pre_allocated < available_bytes) {
+ st = wal_mgr->update_wal_disk_info_map(wal_base_path, -1, -1, pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "update wal disk info map failed, reason: " << st.to_string();
+ }
+ block_queue_pre_allocated.fetch_add(pre_allocated);
+ return true;
+ } else {
+ return false;
+ }
+}
+
+Status GroupCommitMgr::update_load_info(TUniqueId load_id, size_t content_length) {
+ std::unique_lock l(_load_info_lock);
+ if (_load_id_to_content_length_map.find(load_id) != _load_id_to_content_length_map.end()) {
+ return Status::InternalError("load id already exists!");
+ }
+ _load_id_to_content_length_map.insert(std::make_pair(load_id, content_length));
+ return Status::OK();
+}
+
+Status GroupCommitMgr::get_load_info(TUniqueId load_id, size_t* content_length) {
+ std::shared_lock l(_load_info_lock);
+ if (_load_id_to_content_length_map.find(load_id) == _load_id_to_content_length_map.end()) {
+ return Status::InternalError("can not find load id!");
+ }
+ *content_length = _load_id_to_content_length_map[load_id];
+ return Status::OK();
+}
+
+Status GroupCommitMgr::remove_load_info(TUniqueId load_id) {
Review Comment:
warning: method 'remove_load_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:157:
```diff
- Status remove_load_info(TUniqueId load_id);
+ static Status remove_load_info(TUniqueId load_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +473,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return true;
+ }
+ }
+ return false;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->second->available()
+ : std::max_element(_wal_disk_info_map.begin(), _wal_disk_info_map.end(),
+ [](const auto& map1, const auto& map2) {
+ return map1.second->available() <
+ map2.second->available();
+ })
+ ->second->available();
+}
+
+Status WalManager::update_wal_disk_info_map(std::string wal_dir, size_t limit, size_t used,
Review Comment:
warning: method 'update_wal_disk_info_map' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::update_wal_disk_info_map(std::string wal_dir, size_t limit, size_t used,
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -495,4 +505,70 @@ Status LoadBlockQueue::close_wal() {
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
+ const std::vector<std::shared_ptr<vectorized::Block>>& blocks, const TUniqueId& load_id,
+ bool is_blocks_contain_all_load_data) {
+ size_t blocks_size = 0;
+ for (auto block : blocks) {
+ blocks_size += block->bytes();
+ }
+ size_t content_length = 0;
+ Status st = ExecEnv::GetInstance()->group_commit_mgr()->get_load_info(load_id, &content_length);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get load info!";
+ }
+ st = ExecEnv::GetInstance()->group_commit_mgr()->remove_load_info(load_id);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to remove load info!";
+ }
+ size_t pre_allocated = is_blocks_contain_all_load_data
+ ? blocks_size
+ : (blocks_size > content_length ? blocks_size : content_length);
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ size_t available_bytes = 0;
+ {
+ st = wal_mgr->get_wal_disk_available_size(wal_base_path, &available_bytes);
+ if (!st.ok()) {
+ LOG(WARNING) << "get wal disk available size filed!";
+ }
+ }
+ if (pre_allocated < available_bytes) {
+ st = wal_mgr->update_wal_disk_info_map(wal_base_path, -1, -1, pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "update wal disk info map failed, reason: " << st.to_string();
+ }
+ block_queue_pre_allocated.fetch_add(pre_allocated);
+ return true;
+ } else {
+ return false;
+ }
+}
+
+Status GroupCommitMgr::update_load_info(TUniqueId load_id, size_t content_length) {
Review Comment:
warning: method 'update_load_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:155:
```diff
- Status update_load_info(TUniqueId load_id, size_t content_length);
+ static Status update_load_info(TUniqueId load_id, size_t content_length);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -66,6 +72,37 @@ void WalManager::stop() {
}
Status WalManager::init() {
+ RETURN_IF_ERROR(_init_wal_dirs_conf());
+ RETURN_IF_ERROR(_init_wal_dirs());
+ RETURN_IF_ERROR(_init_wal_disk_info());
+ return Thread::create(
+ "WalMgr", "replay_wal", [this]() { static_cast<void>(this->replay()); },
+ &_replay_thread);
+}
+
+Status WalManager::_init_wal_dirs_conf() {
Review Comment:
warning: method '_init_wal_dirs_conf' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:99:
```diff
- Status _init_wal_dirs_conf();
+ static Status _init_wal_dirs_conf();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +473,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
Review Comment:
warning: method 'is_wal_disk_space_enough' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:93:
```diff
- bool is_wal_disk_space_enough();
+ static bool is_wal_disk_space_enough();
```
##########
be/src/runtime/group_commit_mgr.cpp:
##########
@@ -495,4 +505,70 @@
}
return Status::OK();
}
+
+bool LoadBlockQueue::is_wal_disk_space_enough(
+ const std::vector<std::shared_ptr<vectorized::Block>>& blocks, const TUniqueId& load_id,
+ bool is_blocks_contain_all_load_data) {
+ size_t blocks_size = 0;
+ for (auto block : blocks) {
+ blocks_size += block->bytes();
+ }
+ size_t content_length = 0;
+ Status st = ExecEnv::GetInstance()->group_commit_mgr()->get_load_info(load_id, &content_length);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to get load info!";
+ }
+ st = ExecEnv::GetInstance()->group_commit_mgr()->remove_load_info(load_id);
+ if (!st.ok()) {
+ LOG(WARNING) << "failed to remove load info!";
+ }
+ size_t pre_allocated = is_blocks_contain_all_load_data
+ ? blocks_size
+ : (blocks_size > content_length ? blocks_size : content_length);
+ auto* wal_mgr = ExecEnv::GetInstance()->wal_mgr();
+ size_t available_bytes = 0;
+ {
+ st = wal_mgr->get_wal_disk_available_size(wal_base_path, &available_bytes);
+ if (!st.ok()) {
+ LOG(WARNING) << "get wal disk available size filed!";
+ }
+ }
+ if (pre_allocated < available_bytes) {
+ st = wal_mgr->update_wal_disk_info_map(wal_base_path, -1, -1, pre_allocated);
+ if (!st.ok()) {
+ LOG(WARNING) << "update wal disk info map failed, reason: " << st.to_string();
+ }
+ block_queue_pre_allocated.fetch_add(pre_allocated);
+ return true;
+ } else {
+ return false;
+ }
+}
+
+Status GroupCommitMgr::update_load_info(TUniqueId load_id, size_t content_length) {
+ std::unique_lock l(_load_info_lock);
+ if (_load_id_to_content_length_map.find(load_id) != _load_id_to_content_length_map.end()) {
+ return Status::InternalError("load id already exists!");
+ }
+ _load_id_to_content_length_map.insert(std::make_pair(load_id, content_length));
+ return Status::OK();
+}
+
+Status GroupCommitMgr::get_load_info(TUniqueId load_id, size_t* content_length) {
Review Comment:
warning: method 'get_load_info' can be made static [readability-convert-member-functions-to-static]
be/src/runtime/group_commit_mgr.h:156:
```diff
- Status get_load_info(TUniqueId load_id, size_t* content_length);
+ static Status get_load_info(TUniqueId load_id, size_t* content_length);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -345,31 +419,20 @@
}
}
-Status WalManager::delete_wal(int64_t wal_id) {
+Status WalManager::delete_wal(int64_t wal_id, size_t block_queue_pre_allocated) {
Review Comment:
warning: method 'delete_wal' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:63:
```diff
- Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
+ static Status delete_wal(int64_t wal_id, size_t block_queue_pre_allocated = 0);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -410,4 +473,114 @@
return Status::OK();
}
+bool WalManager::is_wal_disk_space_enough() {
+ // if all disks space usage < 80%
+ std::shared_lock l(_wal_disk_info_lock);
+ for (const auto& it : _wal_disk_info_map) {
+ size_t limit = it.second->limit;
+ size_t available = it.second->available();
+ if (available >= limit * 0.8) {
+ return true;
+ }
+ }
+ return false;
+}
+
+const string& WalManager::get_min_disk_usage_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_dirs.size() == 1
+ ? _wal_dirs[0]
+ : *std::min_element(_wal_dirs.begin(), _wal_dirs.end(),
+ [this](const std::string& dir1, const std::string& dir2) {
+ return _wal_disk_info_map[dir1]->available() <
+ _wal_disk_info_map[dir2]->available();
+ });
+}
+
+const string& WalManager::get_random_wal_dir() {
+ std::shared_lock l(_wal_disk_info_lock);
+ return _wal_disk_info_map.size() == 1
+ ? _wal_disk_info_map.begin()->first
+ : std::next(_wal_disk_info_map.begin(), rand() % _wal_disk_info_map.size())
+ ->first;
+}
+
+size_t WalManager::get_max_available_size() {
Review Comment:
warning: method 'get_max_available_size' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:95:
```diff
- size_t get_max_available_size();
+ static size_t get_max_available_size();
```
##########
be/src/io/fs/local_file_system.cpp:
##########
@@ -172,6 +172,19 @@ Status LocalFileSystem::file_size_impl(const Path& file, int64_t* file_size) con
return Status::OK();
}
+Status LocalFileSystem::directory_size(const Path& dir_path, size_t* dir_size) {
Review Comment:
warning: method 'directory_size' can be made static [readability-convert-member-functions-to-static]
be/src/io/fs/local_file_system.h:80:
```diff
- Status directory_size(const Path& dir_path, size_t* dir_size);
+ static Status directory_size(const Path& dir_path, size_t* dir_size);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862613816
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit b0de839d6cd55c382e3a7023835bfffda4f9324a, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4682 4416 4410 4410
q2 370 144 159 144
q3 1444 1291 1247 1247
q4 1113 927 885 885
q5 3174 3131 3151 3131
q6 244 130 125 125
q7 978 491 481 481
q8 2222 2205 2194 2194
q9 6681 6661 6674 6661
q10 3207 3277 3292 3277
q11 315 186 180 180
q12 349 209 212 209
q13 4552 3825 3781 3781
q14 242 212 216 212
q15 568 531 516 516
q16 439 383 383 383
q17 993 568 549 549
q18 7213 6916 6914 6914
q19 1495 1292 1365 1292
q20 508 312 303 303
q21 3112 2606 2702 2606
q22 351 277 287 277
Total cold run time: 44252 ms
Total hot run time: 39777 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4336 4348 4300 4300
q2 269 162 172 162
q3 3549 3539 3523 3523
q4 2397 2386 2379 2379
q5 5763 5769 5772 5769
q6 241 122 122 122
q7 2385 1893 1850 1850
q8 3525 3524 3647 3524
q9 9042 9004 8998 8998
q10 3911 4021 3975 3975
q11 489 366 371 366
q12 761 615 601 601
q13 4314 3583 3600 3583
q14 292 254 251 251
q15 570 523 526 523
q16 522 454 483 454
q17 1867 1853 1851 1851
q18 8622 8215 8274 8215
q19 1734 1743 1737 1737
q20 2277 1954 1941 1941
q21 6558 6226 6189 6189
q22 488 411 417 411
Total cold run time: 63912 ms
Total hot run time: 60724 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27726:
URL: https://github.com/apache/doris/pull/27726#discussion_r1436223214
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
Review Comment:
warning: method 'available' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::available() const {
```
be/src/olap/wal_dirs_info.h:48:
```diff
- size_t available();
+ size_t available() const;
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
Review Comment:
warning: method 'add' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:67:
```diff
- Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
+ static Status add(const std::string& wal_dir, size_t limit, size_t used, size_t pre_allocated);
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
Review Comment:
warning: method 'clear' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:68:
```diff
- Status clear();
+ static Status clear();
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
Review Comment:
warning: method 'get_used' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::get_used() const {
```
be/src/olap/wal_dirs_info.h:43:
```diff
- size_t get_used();
+ size_t get_used() const;
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
Review Comment:
warning: method 'get_available_random_wal_dir' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:69:
```diff
- std::string get_available_random_wal_dir();
+ static std::string get_available_random_wal_dir();
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
Review Comment:
warning: method 'update_all_wal_dir_used' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:74:
```diff
- Status update_all_wal_dir_used();
+ static Status update_all_wal_dir_used();
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_used(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
Review Comment:
warning: method 'update_wal_dir_pre_allocated' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:75:
```diff
- Status update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
+ static Status update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
Review Comment:
warning: method 'update_wal_dir_used' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:73:
```diff
- Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
+ static Status update_wal_dir_used(std::string wal_dir, size_t used = -1);
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
Review Comment:
warning: method 'update_all_wal_dir_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:72:
```diff
- Status update_all_wal_dir_limit();
+ static Status update_all_wal_dir_limit();
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
Review Comment:
warning: method 'update_wal_dir_limit' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:71:
```diff
- Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
+ static Status update_wal_dir_limit(std::string wal_dir, size_t limit = -1);
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
Review Comment:
warning: method 'get_limit' can be made const [readability-make-member-function-const]
```suggestion
size_t WalDirInfo::get_limit() const {
```
be/src/olap/wal_dirs_info.h:42:
```diff
- size_t get_limit();
+ size_t get_limit() const;
```
##########
be/src/olap/wal_dirs_info.cpp:
##########
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/wal_dirs_info.h"
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+
+#include "common/config.h"
+#include "common/status.h"
+#include "io/fs/local_file_system.h"
+#include "util/parse_util.h"
+
+namespace doris {
+
+std::string WalDirInfo::get_wal_dir() {
+ return _wal_dir;
+}
+
+size_t WalDirInfo::get_limit() {
+ std::shared_lock rlock(_lock);
+ return _limit;
+}
+
+size_t WalDirInfo::get_used() {
+ std::shared_lock rlock(_lock);
+ return _used;
+}
+
+size_t WalDirInfo::get_pre_allocated() {
+ std::shared_lock rlock(_lock);
+ return _pre_allocated;
+}
+
+Status WalDirInfo::set_limit(size_t limit) {
+ std::unique_lock wlock(_lock);
+ _limit = limit;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_used(size_t used) {
+ std::unique_lock wlock(_lock);
+ _used = used;
+ return Status::OK();
+}
+
+Status WalDirInfo::set_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ std::unique_lock wlock(_lock);
+ if (is_add_pre_allocated) {
+ _pre_allocated += pre_allocated;
+ } else {
+ _pre_allocated -= pre_allocated;
+ }
+ return Status::OK();
+}
+
+size_t WalDirInfo::available() {
+ std::unique_lock wlock(_lock);
+ int64_t available = _limit - _used - _pre_allocated;
+ return available > 0 ? available : 0;
+}
+
+Status WalDirInfo::update_wal_dir_limit(size_t limit) {
+ if (limit != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_limit(limit));
+ } else {
+ size_t available_bytes;
+ size_t disk_capacity_bytes;
+ RETURN_IF_ERROR(io::global_local_filesystem()->get_space_info(
+ _wal_dir, &disk_capacity_bytes, &available_bytes));
+ bool is_percent = true;
+ int64_t wal_disk_limit = ParseUtil::parse_mem_spec(config::group_commit_wal_max_disk_limit,
+ -1, available_bytes, &is_percent);
+ if (wal_disk_limit <= 0) {
+ return Status::InternalError("Disk full! Please check your disk usage!");
+ }
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_limit(wal_disk_limit));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_used(size_t used) {
+ if (used != static_cast<size_t>(-1)) {
+ RETURN_IF_ERROR(set_used(used));
+ } else {
+ size_t wal_dir_size = 0;
+ RETURN_IF_ERROR(io::global_local_filesystem()->directory_size(_wal_dir, &wal_dir_size));
+ RETURN_IF_ERROR(set_used(wal_dir_size));
+ }
+ return Status::OK();
+}
+
+Status WalDirInfo::update_wal_dir_pre_allocated(size_t pre_allocated, bool is_add_pre_allocated) {
+ RETURN_IF_ERROR(set_pre_allocated(pre_allocated, is_add_pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::add(const std::string& wal_dir, size_t limit, size_t used,
+ size_t pre_allocated) {
+ for (const auto& it : _wal_dirs_info_vec) {
+ if (it->get_wal_dir() == wal_dir) {
+#ifdef BE_TEST
+ return Status::OK();
+#endif
+ return Status::InternalError("wal dir {} exists!", wal_dir);
+ }
+ }
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.emplace_back(
+ std::make_shared<WalDirInfo>(wal_dir, limit, used, pre_allocated));
+ return Status::OK();
+}
+
+Status WalDirsInfo::clear() {
+ std::unique_lock wlock(_lock);
+ _wal_dirs_info_vec.clear();
+ return Status::OK();
+}
+
+std::string WalDirsInfo::get_available_random_wal_dir() {
+ if (_wal_dirs_info_vec.size() == 1) {
+ return (*_wal_dirs_info_vec.begin())->get_wal_dir();
+ } else {
+ std::vector<std::string> available_wal_dirs;
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->available() > wal_dir_info->get_limit() * 0.2) {
+ available_wal_dirs.emplace_back(wal_dir_info->get_wal_dir());
+ }
+ }
+ if (available_wal_dirs.empty()) {
+ return (*std::min_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->get_wal_dir();
+ } else {
+ return (*std::next(_wal_dirs_info_vec.begin(), rand() % _wal_dirs_info_vec.size()))
+ ->get_wal_dir();
+ }
+ }
+}
+
+size_t WalDirsInfo::get_max_available_size() {
+ return _wal_dirs_info_vec.size() == 1
+ ? (*_wal_dirs_info_vec.begin())->available()
+ : (*std::max_element(_wal_dirs_info_vec.begin(), _wal_dirs_info_vec.end(),
+ [](const auto& info1, const auto& info2) {
+ return info1->available() < info2->available();
+ }))
+ ->available();
+}
+
+Status WalDirsInfo::update_wal_dir_limit(std::string wal_dir, size_t limit) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_limit(limit);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_limit() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_limit(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_used(std::string wal_dir, size_t used) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_used(used);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::update_all_wal_dir_used() {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ RETURN_IF_ERROR(wal_dir_info->update_wal_dir_used(-1));
+ }
+ return Status::OK();
+}
+
+Status WalDirsInfo::update_wal_dir_pre_allocated(std::string wal_dir, size_t pre_allocated,
+ bool is_add_pre_allocated) {
+ for (const auto& wal_dir_info : _wal_dirs_info_vec) {
+ if (wal_dir_info->get_wal_dir() == wal_dir) {
+ return wal_dir_info->update_wal_dir_pre_allocated(pre_allocated, is_add_pre_allocated);
+ }
+ }
+ return Status::InternalError("Can not find wal dir in wal disks info.");
+}
+
+Status WalDirsInfo::get_wal_dir_available_size(const std::string& wal_dir,
Review Comment:
warning: method 'get_wal_dir_available_size' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_dirs_info.h:77:
```diff
- Status get_wal_dir_available_size(const std::string& wal_dir, size_t* available_bytes);
+ static Status get_wal_dir_available_size(const std::string& wal_dir, size_t* available_bytes);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #27726:
URL: https://github.com/apache/doris/pull/27726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1862432740
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858738502
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.26 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17222637243 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1858797639
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.12 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17219580710 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27726:
URL: https://github.com/apache/doris/pull/27726#issuecomment-1859487838
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.35 seconds
stream load tsv: 591 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17221293883 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org