You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/11/12 10:14:35 UTC
[GitHub] [incubator-doris] weizuo93 opened a new issue #4882: [Proposal] Execute 'pick rowsets' before applying for permits for a compaction task
weizuo93 opened a new issue #4882:
URL: https://github.com/apache/incubator-doris/issues/4882
The current compaction mechanism is that there is a producer thread that has been producing compaction tasks, and the selected tablet must apply for `permits`. When a tablet could hold `permits`, compaction task for this tablet will be submitted to thread pool. We take compaction score as `permits` which is used for limiting memory consumption. However, `pick_rowset_to_compaction()` will be executed before the file merge in compaction thread, and the number of segment files that actually perform the merge operation is smaller than compaction score. In addition, it is also possible that compaction task exits directly because the tablet doesn't meet the requirements of compaction.
Can we execute 'pick rowsets' before applying for permits for a compaction task, calculate the number of segment files that actually participate in the merge operation, and take this number as `permits`.
Based on test results, we found that`pick_rowset_to_compaction()` execution time for a tablet is on the time scale of milliseconds. Therefore, there is little pressure on the producer thread.
```
1112 16:00:08.686122 (+ 982us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:08.355522 (+ 1033us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:08.355372 (+ 930us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:09.707385 (+ 990us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.218772 (+ 1305us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.212823 (+ 1212us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:32.077322 (+ 2411us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:33.534394 (+ 2484us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:52.350023 (+ 1181us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:54.011435 (+ 1212us) cumulative_compaction.cpp:50] rowsets picked
1112 16:01:14.845957 (+ 294us) cumulative_compaction.cpp:50] rowsets picked
1112 16:00:17.865887 (+ 97us) base_compaction.cpp:44] rowsets picked
1112 16:01:35.193204 (+ 206us) cumulative_compaction.cpp:50] rowsets picked
1112 16:01:39.244388 (+ 198us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:18.931984 (+ 1627us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:19.014592 (+ 1394us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:03.405753 (+ 160us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:04.777374 (+ 350us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:07.016017 (+ 164us) cumulative_compaction.cpp:50] rowsets picked
1112 15:59:12.573256 (+ 205us) base_compaction.cpp:44] rowsets picked
1112 16:02:18.932822 (+ 2213us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:08.975090 (+ 339us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:18.932387 (+ 1885us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:11.285206 (+ 402us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:14.799639 (+ 119us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:20.540975 (+ 148us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:23.908919 (+ 198us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:32.000026 (+ 133us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:13.347364 (+ 2796us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:13.722429 (+ 2046us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:43.050516 (+ 410us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:43.849807 (+ 387us) cumulative_compaction.cpp:50] rowsets picked
1112 16:02:17.914220 (+ 469us) base_compaction.cpp:44] rowsets picked
1112 16:03:48.663824 (+ 336us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:49.778246 (+ 241us) cumulative_compaction.cpp:50] rowsets picked
1112 16:03:52.141355 (+ 468us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:20.670042 (+ 1564us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:38.078296 (+ 264us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:36.673882 (+ 1177us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:37.060312 (+ 1139us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:55.154987 (+ 1215us) cumulative_compaction.cpp:50] rowsets picked
1112 16:04:55.403399 (+ 1071us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:02.536426 (+ 1551us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:32.090439 (+ 1024us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:18.661826 (+ 1306us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:18.990078 (+ 1241us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:42.798424 (+ 143us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:46.082527 (+ 176us) cumulative_compaction.cpp:50] rowsets picked
1112 16:05:41.727956 (+ 1584us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:01.083588 (+ 1123us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:26.622230 (+ 632us) cumulative_compaction.cpp:50] rowsets picked
1112 16:06:35.897463 (+ 92us) cumulative_compaction.cpp:50] rowsets picked
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #4882: [Proposal] Execute 'pick rowsets' before applying for permits for a compaction task
Posted by GitBox <gi...@apache.org>.
morningman closed issue #4882:
URL: https://github.com/apache/incubator-doris/issues/4882
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org