You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liaoxin01 (via GitHub)" <gi...@apache.org> on 2023/12/05 08:03:22 UTC
[PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
liaoxin01 opened a new pull request, #28012:
URL: https://github.com/apache/doris/pull/28012
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
1. Do not retry publishing on be When there are too many missing versions, just add to async publish task.
2. To reduce memory consumption, clean up the tasks when there are too many async publish tasks.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #28012:
URL: https://github.com/apache/doris/pull/28012#discussion_r1423321938
##########
be/src/olap/task/engine_publish_version_task.cpp:
##########
@@ -191,8 +191,16 @@ Status EnginePublishVersionTask::execute() {
}
auto handle_version_not_continuous = [&]() {
add_error_tablet_id(tablet_info.tablet_id);
- _discontinuous_version_tablets->emplace_back(
- partition_id, tablet_info.tablet_id, version.first);
+ // When there are too many missing versions, do not directly retry the
+ // publish and handle it through async publish.
+ if (max_version + 20 < version.first) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen commented on code in PR #28012:
URL: https://github.com/apache/doris/pull/28012#discussion_r1418619022
##########
be/src/olap/task/engine_publish_version_task.cpp:
##########
@@ -191,8 +191,16 @@ Status EnginePublishVersionTask::execute() {
}
auto handle_version_not_continuous = [&]() {
add_error_tablet_id(tablet_info.tablet_id);
- _discontinuous_version_tablets->emplace_back(
- partition_id, tablet_info.tablet_id, version.first);
+ // When there are too many missing versions, do not directly retry the
+ // publish and handle it through async publish.
+ if (max_version + 20 < version.first) {
Review Comment:
We'd better make this number configurable so that in case there's an issue with this strategy, users can adjust the configuration to work around it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1851437485
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1851259805
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit bdb6599878ee02b3038b6c7c427fb1ad91646bbe, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4695 4455 4479 4455
q2 368 151 151 151
q3 1447 1258 1277 1258
q4 1114 919 929 919
q5 3156 3123 3128 3123
q6 252 126 128 126
q7 990 490 485 485
q8 2230 2247 2192 2192
q9 6668 6675 6656 6656
q10 3192 3237 3241 3237
q11 323 206 209 206
q12 341 198 205 198
q13 4535 3802 3790 3790
q14 254 208 216 208
q15 565 528 522 522
q16 450 386 387 386
q17 1013 553 562 553
q18 7526 7319 8326 7319
q19 1512 1386 1479 1386
q20 498 722 313 313
q21 3070 2666 2683 2666
q22 361 282 288 282
Total cold run time: 44560 ms
Total hot run time: 40431 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4384 4407 4397 4397
q2 272 166 176 166
q3 3553 3549 3537 3537
q4 2389 2380 2368 2368
q5 5750 5732 5763 5732
q6 237 122 122 122
q7 2401 1870 1937 1870
q8 3513 3540 3516 3516
q9 9105 9047 9021 9021
q10 3889 3991 3977 3977
q11 506 378 371 371
q12 772 606 584 584
q13 4296 3571 3543 3543
q14 291 250 270 250
q15 567 526 520 520
q16 493 462 473 462
q17 1896 1863 1849 1849
q18 8817 8382 8324 8324
q19 1725 1771 1771 1771
q20 2257 1943 1953 1943
q21 6547 6176 6232 6176
q22 509 425 419 419
Total cold run time: 64169 ms
Total hot run time: 60918 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1840531873
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.44 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17163826226 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1840531965
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 5896476d7843fb3036a61930f69eb45906631597, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4726 4461 4461 4461
q2 361 155 158 155
q3 1466 1268 1200 1200
q4 1125 955 932 932
q5 3223 3163 3203 3163
q6 246 133 130 130
q7 984 489 493 489
q8 2238 2225 2196 2196
q9 6663 6680 6630 6630
q10 3193 3232 3223 3223
q11 316 199 196 196
q12 355 214 217 214
q13 4546 3785 3788 3785
q14 240 212 224 212
q15 568 522 519 519
q16 436 377 383 377
q17 1006 628 570 570
q18 7461 7494 6868 6868
q19 1522 1455 1393 1393
q20 569 735 323 323
q21 3125 2742 2710 2710
q22 356 290 292 290
Total cold run time: 44725 ms
Total hot run time: 40036 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4392 4364 4378 4364
q2 269 161 177 161
q3 3522 3570 3505 3505
q4 2379 2390 2364 2364
q5 5744 5750 5719 5719
q6 237 121 120 120
q7 2359 1836 1836 1836
q8 3521 3522 3527 3522
q9 9024 8980 9050 8980
q10 3905 4002 3997 3997
q11 511 389 394 389
q12 759 609 607 607
q13 4303 3596 3576 3576
q14 282 247 243 243
q15 564 517 515 515
q16 496 465 452 452
q17 1864 1821 1848 1821
q18 8627 8307 8303 8303
q19 1756 1775 1768 1768
q20 2279 1958 1950 1950
q21 6487 6210 6136 6136
q22 495 423 414 414
Total cold run time: 63775 ms
Total hot run time: 60742 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "zhannngchen (via GitHub)" <gi...@apache.org>.
zhannngchen merged PR #28012:
URL: https://github.com/apache/doris/pull/28012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1851269039
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.72 seconds
stream load tsv: 593 seconds loaded 74807831229 Bytes, about 120 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17216555798 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1840210739
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1851437543
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #28012:
URL: https://github.com/apache/doris/pull/28012#issuecomment-1851190894
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](merge-on-write) Optimize publish when there are missing versions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28012:
URL: https://github.com/apache/doris/pull/28012#discussion_r1415088370
##########
be/src/olap/olap_server.cpp:
##########
@@ -1221,58 +1236,67 @@ int64_t StorageEngine::get_pending_publish_min_version(int64_t tablet_id) {
return iter->second.begin()->first;
}
-void StorageEngine::_async_publish_callback() {
- while (!_stop_background_threads_latch.wait_for(std::chrono::milliseconds(30))) {
- // tablet, publish_version
- std::vector<std::pair<TabletSharedPtr, int64_t>> need_removed_tasks;
- {
- std::lock_guard<std::mutex> lock(_async_publish_mutex);
- for (auto tablet_iter = _async_publish_tasks.begin();
- tablet_iter != _async_publish_tasks.end();) {
- if (tablet_iter->second.empty()) {
- tablet_iter = _async_publish_tasks.erase(tablet_iter);
- continue;
- }
- int64_t tablet_id = tablet_iter->first;
- TabletSharedPtr tablet = tablet_manager()->get_tablet(tablet_id);
- if (!tablet) {
- LOG(WARNING) << "tablet does not exist when async publush, tablet_id: "
- << tablet_id;
- tablet_iter = _async_publish_tasks.erase(tablet_iter);
- continue;
- }
+void StorageEngine::_process_async_publish() {
Review Comment:
warning: method '_process_async_publish' can be made static [readability-convert-member-functions-to-static]
be/src/olap/storage_engine.h:336:
```diff
- void _process_async_publish();
+ static void _process_async_publish();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org