You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2023/12/12 15:36:15 UTC
[PR] [WIP] relay wal p0 test [doris]
hust-hhb opened a new pull request, #28299:
URL: https://github.com/apache/doris/pull/28299
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857868249
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit fa9ba9ec7bf1636213f61279009e86179502c612, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4904 4656 4710 4656
q2 370 143 158 143
q3 1509 1317 1271 1271
q4 1157 941 974 941
q5 3254 3264 3225 3225
q6 259 129 128 128
q7 988 506 545 506
q8 2258 2306 2281 2281
q9 6884 6857 6824 6824
q10 3231 3327 3336 3327
q11 341 201 208 201
q12 383 210 207 207
q13 4616 3848 3781 3781
q14 250 217 217 217
q15 569 523 531 523
q16 455 407 395 395
q17 1017 608 583 583
q18 7131 6797 6846 6797
q19 1592 1533 1419 1419
q20 562 311 303 303
q21 3214 2822 2831 2822
q22 361 279 290 279
Total cold run time: 45305 ms
Total hot run time: 40829 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4622 4652 4609 4609
q2 291 169 177 169
q3 3599 3576 3583 3576
q4 2469 2469 2457 2457
q5 5848 5845 5853 5845
q6 258 123 123 123
q7 2391 1866 1856 1856
q8 3638 3643 3645 3643
q9 9006 8965 8912 8912
q10 3972 4058 4065 4058
q11 527 396 392 392
q12 777 596 585 585
q13 4319 3553 3596 3553
q14 292 248 259 248
q15 575 528 510 510
q16 515 451 461 451
q17 1957 1942 1906 1906
q18 8717 8279 8225 8225
q19 1825 1833 1863 1833
q20 2266 1943 1934 1934
q21 6690 6295 6328 6295
q22 517 420 428 420
Total cold run time: 65071 ms
Total hot run time: 61600 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857324219
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 8dc8f160476071931ff343898229b023185611a1, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4702 4456 4448 4448
q2 358 146 138 138
q3 1471 1225 1227 1225
q4 1105 930 862 862
q5 3164 3164 3168 3164
q6 241 127 130 127
q7 990 490 476 476
q8 2206 2231 2213 2213
q9 6673 6664 6723 6664
q10 3220 3288 3275 3275
q11 326 204 193 193
q12 350 204 212 204
q13 4541 3788 3776 3776
q14 242 216 220 216
q15 570 526 524 524
q16 446 386 387 386
q17 1030 610 578 578
q18 7125 6916 6890 6890
q19 1521 1450 1358 1358
q20 503 335 282 282
q21 3027 2622 2614 2614
q22 350 281 294 281
Total cold run time: 44161 ms
Total hot run time: 39894 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4406 4327 4372 4327
q2 272 165 170 165
q3 3521 3507 3494 3494
q4 2389 2371 2371 2371
q5 5730 5746 5719 5719
q6 239 119 123 119
q7 2379 1876 1844 1844
q8 3522 3520 3527 3520
q9 8976 9008 9035 9008
q10 3933 4007 4029 4007
q11 494 387 391 387
q12 764 584 594 584
q13 4303 3585 3562 3562
q14 287 260 249 249
q15 574 523 522 522
q16 526 466 488 466
q17 1882 1857 1845 1845
q18 8704 8182 8157 8157
q19 1760 1747 1767 1747
q20 2239 1948 1926 1926
q21 6490 6162 6138 6138
q22 489 427 415 415
Total cold run time: 63879 ms
Total hot run time: 60572 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1858818285
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.11 seconds
stream load tsv: 611 seconds loaded 74807831229 Bytes, about 116 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17219312193 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1854268708
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 1a594f3ddc881597e1ee102e9e0f5f1444f500f0, data reload: false
run tpch-sf100 query with default conf and session variables
q1 50408 31854 31866 31854
q2 365 135 145 135
q3 1485 1257 1237 1237
q4 1107 926 883 883
q5 3158 3160 3158 3158
q6 259 138 153 138
q7 984 492 478 478
q8 2148 2215 2200 2200
q9 6944 6903 6861 6861
q10 3247 3334 3322 3322
q11 339 201 214 201
q12 365 215 206 206
q13 4575 3793 3784 3784
q14 429 318 317 317
q15 800 1315 1331 1315
q16 434 392 386 386
q17 1015 609 586 586
q18 7355 7635 7226 7226
q19 1538 1468 1392 1392
q20 609 301 296 296
q21 3060 2673 2615 2615
q22 352 283 292 283
Total cold run time: 90976 ms
Total hot run time: 68873 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30832 31865 31827 31827
q2 278 163 175 163
q3 3644 3596 3581 3581
q4 2405 2376 2378 2376
q5 5773 5752 5767 5752
q6 243 136 138 136
q7 3027 3419 3371 3371
q8 3538 3539 3536 3536
q9 9237 9199 9141 9141
q10 3958 4068 4072 4068
q11 802 908 1366 908
q12 765 581 598 581
q13 4327 3604 3541 3541
q14 503 430 430 430
q15 1151 1358 1331 1331
q16 499 464 493 464
q17 1894 1868 1858 1858
q18 8784 8186 8094 8094
q19 1801 1779 1769 1769
q20 2940 2592 2666 2592
q21 6554 6149 6125 6125
q22 508 433 422 422
Total cold run time: 93463 ms
Total hot run time: 92066 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1856519501
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 267e0eba0e4c93b5d5a2d347c246d22c89bb7553, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4743 4431 4474 4431
q2 369 146 155 146
q3 1525 1242 1248 1242
q4 1125 900 915 900
q5 3166 3162 3162 3162
q6 251 132 127 127
q7 996 492 491 491
q8 2189 2212 2189 2189
q9 6650 6702 6652 6652
q10 3215 3243 3253 3243
q11 326 207 209 207
q12 346 205 207 205
q13 4559 3825 3804 3804
q14 236 215 209 209
q15 569 523 520 520
q16 435 389 392 389
q17 1006 563 578 563
q18 7291 6854 6904 6854
q19 1510 1379 1423 1379
q20 548 285 314 285
q21 3087 2628 2647 2628
q22 350 282 285 282
Total cold run time: 44492 ms
Total hot run time: 39908 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4406 4389 4406 4389
q2 266 164 167 164
q3 3531 3540 3529 3529
q4 2384 2364 2356 2356
q5 5717 5752 5732 5732
q6 241 122 120 120
q7 2401 1860 1827 1827
q8 3519 3512 3523 3512
q9 9046 8906 8970 8906
q10 3904 3972 3971 3971
q11 492 379 384 379
q12 761 587 585 585
q13 4273 3607 3598 3598
q14 288 260 243 243
q15 565 529 520 520
q16 501 490 489 489
q17 1893 1876 1842 1842
q18 8680 8152 8243 8152
q19 1730 1735 1744 1735
q20 2250 1951 1936 1936
q21 6545 6168 6143 6143
q22 496 454 434 434
Total cold run time: 63889 ms
Total hot run time: 60562 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28299:
URL: https://github.com/apache/doris/pull/28299#discussion_r1427102309
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +412,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
Review Comment:
warning: method 'notify' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:81:
```diff
- Status notify(int64_t wal_id);
+ static Status notify(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +412,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
Review Comment:
warning: method 'find_wal_path' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:80:
```diff
- bool find_wal_path(int64_t wal_id);
+ static bool find_wal_path(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +412,80 @@ Status WalManager::get_wal_column_index(int64_t wal_id, std::vector<size_t>& col
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
Review Comment:
warning: method 'erase_wal_cv_map' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:76:
```diff
- Status erase_wal_cv_map(int64_t wal_id);
+ static Status erase_wal_cv_map(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +412,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ LOG(INFO) << "find wal path " << it->second;
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/olap/wal_manager.cpp:481:
```diff
- if (it != _wal_path_map.end()) {
- LOG(INFO) << "find wal path " << it->second;
- return true;
- } else {
- LOG(INFO) << "not find wal path for " << wal_id;
- return false;
- }
+ return it != _wal_path_map.end();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1859073136
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1853850948
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1853923260
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1854515582
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1856383266
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28299:
URL: https://github.com/apache/doris/pull/28299#discussion_r1427632325
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
Review Comment:
warning: method 'notify' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:81:
```diff
- Status notify(int64_t wal_id);
+ static Status notify(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
Review Comment:
warning: method 'get_lock_and_cv' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
Review Comment:
warning: method 'erase_wal_cv_map' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:76:
```diff
- Status erase_wal_cv_map(int64_t wal_id);
+ static Status erase_wal_cv_map(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ LOG(INFO) << "find wal path " << it->second;
Review Comment:
warning: method 'find_wal_path' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:80:
```diff
- bool find_wal_path(int64_t wal_id);
+ static bool find_wal_path(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@ Status WalManager::get_wal_column_index(int64_t wal_id, std::vector<size_t>& col
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
Review Comment:
warning: method 'add_wal_cv_map' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ LOG(INFO) << "find wal path " << it->second;
+ return true;
+ } else {
+ LOG(INFO) << "not find wal path for " << wal_id;
+ return false;
+ }
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/olap/wal_manager.cpp:489:
```diff
- if (it != _wal_path_map.end()) {
- LOG(INFO) << "find wal path " << it->second;
- return true;
- } else {
- LOG(INFO) << "not find wal path for " << wal_id;
- return false;
- }
+ return it != _wal_path_map.end();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857884657
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.99 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17219668632 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1858794145
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1859082472
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit d0bd953843db377a1d5dd0d09dbe130c9ce34f3e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4699 4413 4421 4413
q2 367 139 133 133
q3 1442 1214 1199 1199
q4 1094 924 873 873
q5 3147 3159 3126 3126
q6 249 126 126 126
q7 970 477 489 477
q8 2213 2208 2179 2179
q9 6672 6679 6684 6679
q10 3210 3283 3267 3267
q11 334 201 208 201
q12 346 212 213 212
q13 4529 3761 3761 3761
q14 239 213 207 207
q15 563 519 522 519
q16 441 390 388 388
q17 1006 592 487 487
q18 7230 6813 6922 6813
q19 1531 1438 1363 1363
q20 552 317 288 288
q21 3060 2656 2620 2620
q22 342 274 285 274
Total cold run time: 44236 ms
Total hot run time: 39605 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4367 4385 4387 4385
q2 270 163 170 163
q3 3522 3520 3520 3520
q4 2368 2359 2367 2359
q5 5731 5721 5729 5721
q6 244 120 121 120
q7 2403 1879 1905 1879
q8 3540 3519 3515 3515
q9 9012 8969 8949 8949
q10 3915 3979 4015 3979
q11 494 383 386 383
q12 761 608 601 601
q13 4262 3531 3525 3525
q14 279 256 250 250
q15 566 516 524 516
q16 524 456 460 456
q17 1847 1854 1844 1844
q18 8730 8140 8256 8140
q19 1717 1746 1734 1734
q20 2246 1943 1934 1934
q21 6520 6151 6124 6124
q22 491 425 427 425
Total cold run time: 63809 ms
Total hot run time: 60522 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857382349
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb closed pull request #28299: [WIP] relay wal p0 test
URL: https://github.com/apache/doris/pull/28299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857433157
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.23 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17219415299 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1856535594
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.52 seconds
stream load tsv: 591 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17218104684 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1852487603
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 7fa8b01402c4ba83acb817bab2c8b121a96174f6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 49774 31851 31786 31786
q2 390 140 140 140
q3 1611 1324 1231 1231
q4 1107 931 933 931
q5 3223 3202 3240 3202
q6 267 142 143 142
q7 993 495 490 490
q8 2143 2214 2211 2211
q9 6928 6875 6977 6875
q10 3230 3310 3303 3303
q11 341 223 216 216
q12 353 206 204 204
q13 4579 4042 3828 3828
q14 433 318 329 318
q15 752 1308 1336 1308
q16 451 391 387 387
q17 1016 563 541 541
q18 7656 7196 6952 6952
q19 1521 1360 1443 1360
q20 594 298 293 293
q21 3051 2655 2675 2655
q22 349 283 286 283
Total cold run time: 90762 ms
Total hot run time: 68656 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30788 31825 31792 31792
q2 276 166 175 166
q3 3686 3627 3585 3585
q4 2396 2374 2373 2373
q5 5795 5805 5810 5805
q6 242 131 132 131
q7 3061 3391 3373 3373
q8 3521 3528 3527 3527
q9 9305 9162 9169 9162
q10 3978 4064 4049 4049
q11 821 909 1339 909
q12 778 604 602 602
q13 4312 3595 3557 3557
q14 501 440 432 432
q15 1119 1305 1344 1305
q16 497 449 474 449
q17 1892 1850 1841 1841
q18 8775 8317 8333 8317
q19 1778 1790 1770 1770
q20 2898 2603 2591 2591
q21 6534 6177 6158 6158
q22 515 437 421 421
Total cold run time: 93468 ms
Total hot run time: 92315 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1854143550
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.94 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17221975165 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1852283255
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857384975
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1854569403
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.07 seconds
stream load tsv: 583 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17222472451 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1858815626
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 12d85828078461a55d87cd44a204d22f9d14d6df, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4737 4493 4509 4493
q2 362 151 158 151
q3 1460 1237 1224 1224
q4 1112 906 904 904
q5 3147 3112 3112 3112
q6 245 129 131 129
q7 1004 482 479 479
q8 2215 2192 2177 2177
q9 6712 6641 6694 6641
q10 3193 3281 3276 3276
q11 330 205 204 204
q12 352 213 210 210
q13 4553 3798 3808 3798
q14 236 208 213 208
q15 566 532 521 521
q16 440 379 382 379
q17 1012 563 532 532
q18 7201 7008 6936 6936
q19 1528 1438 1342 1342
q20 541 315 311 311
q21 3110 2673 2682 2673
q22 349 278 283 278
Total cold run time: 44405 ms
Total hot run time: 39978 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4402 4402 4416 4402
q2 268 165 175 165
q3 3532 3520 3503 3503
q4 2382 2391 2365 2365
q5 5736 5720 5734 5720
q6 241 123 124 123
q7 2399 1868 1880 1868
q8 3508 3532 3505 3505
q9 9058 9004 8977 8977
q10 3887 3992 3990 3990
q11 505 397 384 384
q12 764 609 583 583
q13 4284 3579 3541 3541
q14 284 258 255 255
q15 566 513 525 513
q16 510 464 476 464
q17 1884 1848 1865 1848
q18 8573 8247 8139 8139
q19 1740 1763 1741 1741
q20 2249 1947 1924 1924
q21 6507 6172 6146 6146
q22 494 437 430 430
Total cold run time: 63773 ms
Total hot run time: 60586 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857289466
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.14 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17222640671 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28299:
URL: https://github.com/apache/doris/pull/28299#discussion_r1424204914
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(10)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
Review Comment:
warning: method 'notify' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:80:
```diff
- Status notify(int64_t wal_id);
+ static Status notify(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
Review Comment:
warning: method 'erase_wal_cv_map' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:75:
```diff
- Status erase_wal_cv_map(int64_t wal_id);
+ static Status erase_wal_cv_map(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@ Status WalManager::get_wal_column_index(int64_t wal_id, std::vector<size_t>& col
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
Review Comment:
warning: method 'add_wal_cv_map' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(10)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
Review Comment:
warning: method 'find_wal_path' can be made static [readability-convert-member-functions-to-static]
be/src/olap/wal_manager.h:79:
```diff
- bool find_wal_path(int64_t wal_id);
+ static bool find_wal_path(int64_t wal_id);
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(10)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ LOG(INFO) << "find wal path " << it->second;
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/olap/wal_manager.cpp:474:
```diff
- if (it != _wal_path_map.end()) {
- LOG(INFO) << "find wal path " << it->second;
- return true;
- } else {
- LOG(INFO) << "not find wal path for " << wal_id;
- return false;
- }
+ return it != _wal_path_map.end();
```
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +405,80 @@
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(10)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
Review Comment:
warning: method 'get_lock_and_cv' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
```
##########
be/src/runtime/runtime_state.h:
##########
@@ -255,6 +255,10 @@ class RuntimeState {
int64_t wal_id() { return _wal_id; }
+ void set_txn_id(int64_t txn_id) { _txn_id = txn_id; }
+
+ int64_t txn_id() { return _txn_id; }
Review Comment:
warning: method 'txn_id' can be made const [readability-make-member-function-const]
```suggestion
int64_t txn_id() const { return _txn_id; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1852512503
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.39 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17220911318 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1854551876
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit be7a308442834c595694d15a5907cbfb8cb981b2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 51502 31871 31876 31871
q2 362 134 142 134
q3 1501 1230 1202 1202
q4 1097 957 885 885
q5 3191 3220 3207 3207
q6 265 141 153 141
q7 1015 490 490 490
q8 2201 2242 2208 2208
q9 6908 6866 6850 6850
q10 3257 3332 3283 3283
q11 346 219 212 212
q12 352 204 198 198
q13 4565 3834 3767 3767
q14 425 321 314 314
q15 788 1323 1333 1323
q16 444 391 390 390
q17 1029 612 565 565
q18 7222 6795 8355 6795
q19 1581 1474 1364 1364
q20 586 300 293 293
q21 3081 2672 2696 2672
q22 342 275 276 275
Total cold run time: 92060 ms
Total hot run time: 68439 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 30851 31863 31836 31836
q2 279 165 169 165
q3 3617 3611 3583 3583
q4 2393 2386 2375 2375
q5 5777 5771 5756 5756
q6 239 132 136 132
q7 3054 3395 3368 3368
q8 3549 3564 3559 3559
q9 9163 9165 9124 9124
q10 3952 4068 4048 4048
q11 803 928 1346 928
q12 765 585 584 584
q13 4301 3524 3553 3524
q14 518 434 438 434
q15 1138 1352 1338 1338
q16 512 443 464 443
q17 1884 1863 1885 1863
q18 8755 8092 8336 8092
q19 1791 1796 1813 1796
q20 2959 2585 2672 2585
q21 6524 6157 6125 6125
q22 507 412 439 412
Total cold run time: 93331 ms
Total hot run time: 92070 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857259617
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #28299:
URL: https://github.com/apache/doris/pull/28299#issuecomment-1857755080
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP] relay wal p0 test [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28299:
URL: https://github.com/apache/doris/pull/28299#discussion_r1427539769
##########
be/src/olap/wal_manager.cpp:
##########
@@ -392,4 +416,80 @@ Status WalManager::get_wal_column_index(int64_t wal_id, std::vector<size_t>& col
return Status::OK();
}
+Status WalManager::add_wal_cv_map(int64_t wal_id, std::shared_ptr<std::mutex> lock,
+ std::shared_ptr<std::condition_variable> cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto it = _wal_lock_map.find(wal_id);
+ if (it != _wal_lock_map.end()) {
+ return Status::InternalError("wal {} is already in _wal_cv_map ", wal_id);
+ }
+ _wal_lock_map.emplace(wal_id, lock);
+ _wal_cv_map.emplace(wal_id, cv);
+ LOG(INFO) << "add " << wal_id << " to _wal_cv_map";
+ return Status::OK();
+}
+Status WalManager::erase_wal_cv_map(int64_t wal_id) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ if (_wal_lock_map.erase(wal_id) && _wal_cv_map.erase(wal_id)) {
+ LOG(INFO) << "erase " << wal_id << " from _wal_cv_map";
+ } else {
+ return Status::InternalError("fail to erase wal {} from wal_cv_map", wal_id);
+ }
+ return Status::OK();
+}
+Status WalManager::wait_relay_wal_finish(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ LOG(INFO) << "start wait " << wal_id;
+ if (cv->wait_for(l, std::chrono::seconds(180)) == std::cv_status::timeout) {
+ LOG(WARNING) << "wait for " << wal_id << " is time out";
+ }
+ LOG(INFO) << "get wal " << wal_id << ",finish wait";
+ RETURN_IF_ERROR(erase_wal_cv_map(wal_id));
+ LOG(INFO) << "erase wal " << wal_id;
+ }
+ return Status::OK();
+}
+Status WalManager::notify(int64_t wal_id) {
+ std::shared_ptr<std::mutex> lock = nullptr;
+ std::shared_ptr<std::condition_variable> cv = nullptr;
+ auto st = get_lock_and_cv(wal_id, lock, cv);
+ if (st.ok()) {
+ std::unique_lock l(*(lock));
+ cv->notify_all();
+ LOG(INFO) << "get wal " << wal_id << ",notify all";
+ }
+ return Status::OK();
+}
+Status WalManager::get_lock_and_cv(int64_t wal_id, std::shared_ptr<std::mutex>& lock,
+ std::shared_ptr<std::condition_variable>& cv) {
+ std::lock_guard<std::shared_mutex> wrlock(_wal_cv_lock);
+ auto lock_it = _wal_lock_map.find(wal_id);
+ if (lock_it == _wal_lock_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_lock_map", wal_id);
+ }
+ lock = lock_it->second;
+ auto cv_it = _wal_cv_map.find(wal_id);
+ if (cv_it == _wal_cv_map.end()) {
+ return Status::InternalError("cannot find txn {} in wal_cv_map", wal_id);
+ }
+ cv = cv_it->second;
+ return Status::OK();
+}
+
+bool WalManager::find_wal_path(int64_t wal_id) {
+ std::shared_lock rdlock(_wal_lock);
+ auto it = _wal_path_map.find(wal_id);
+ if (it != _wal_path_map.end()) {
+ LOG(INFO) << "find wal path " << it->second;
+ return true;
Review Comment:
warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
be/src/olap/wal_manager.cpp:485:
```diff
- if (it != _wal_path_map.end()) {
- LOG(INFO) << "find wal path " << it->second;
- return true;
- } else {
- LOG(INFO) << "not find wal path for " << wal_id;
- return false;
- }
+ return it != _wal_path_map.end();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org