You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/10 03:47:45 UTC
[GitHub] [incubator-doris] acelyc111 opened a new pull request #3814: [trace] Adapt trace util to compaction module
acelyc111 opened a new pull request #3814:
URL: https://github.com/apache/incubator-doris/pull/3814
Trace util is helpful for diagnosing compaction performance problems,
we can get trace log for base compaction like:
```
W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
0610 11:23:03.727535 (+ 0us) storage_engine.cpp:554] start to perform base compaction
0610 11:23:03.728961 (+ 1426us) storage_engine.cpp:560] found best tablet 546859
0610 11:23:03.728963 (+ 2us) base_compaction.cpp:40] got base compaction lock
0610 11:23:03.729029 (+ 66us) base_compaction.cpp:44] rowsets picked
0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
0610 11:24:51.784818 (+ 379us) compaction.cpp:74] prepare finished
0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
0610 11:26:33.484482 (+ 1us) compaction.cpp:106] check correctness finished
0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
0610 11:26:33.513300 (+ 103us) base_compaction.cpp:49] compaction finished
0610 11:26:33.513441 (+ 141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
```
for cumulative compaction like:
```
W0610 11:14:18.714366 56468 storage_engine.cpp:518] Trace:
0610 11:14:08.068484 (+ 0us) storage_engine.cpp:520] start to perform cumulative compaction
0610 11:14:08.069844 (+ 1360us) storage_engine.cpp:526] found best tablet 547083
0610 11:14:08.069846 (+ 2us) cumulative_compaction.cpp:42] got cumulative compaction lock
0610 11:14:08.069947 (+ 101us) cumulative_compaction.cpp:46] calculated cumulative point
0610 11:14:08.070141 (+ 194us) cumulative_compaction.cpp:50] rowsets picked
0610 11:14:08.070143 (+ 2us) compaction.cpp:46] got concurrency lock and start to do compaction
0610 11:14:08.070518 (+ 375us) compaction.cpp:74] prepare finished
0610 11:14:15.389893 (+7319375us) compaction.cpp:87] merge rowsets finished
0610 11:14:15.390916 (+ 1023us) compaction.cpp:102] output rowset built
0610 11:14:15.390917 (+ 1us) compaction.cpp:106] check correctness finished
0610 11:14:15.409460 (+ 18543us) compaction.cpp:110] modify rowsets finished
0610 11:14:15.409496 (+ 36us) cumulative_compaction.cpp:55] compaction finished
0610 11:14:15.410138 (+ 642us) cumulative_compaction.cpp:65] unused rowsets have been moved to GC queue
Metrics: {"filtered_rows":0,"input_row_num":136707,"input_rowsets_count":302,"input_rowsets_data_size":76617836,"input_segments_num":302,"merge_rowsets_latency_us":7319372,"merged_rows":0,"output_row_num":136707,"output_rowset_data_size":53893280,"output_segments_num":1}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3814: [trace] Adapt trace util to compaction module
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3814:
URL: https://github.com/apache/incubator-doris/pull/3814#discussion_r438186596
##########
File path: be/src/common/config.h
##########
@@ -281,6 +281,10 @@ namespace config {
// This config can be set to 0, which means to forbid any compaction, for some special cases.
CONF_Int32(max_compaction_concurrency, "-1");
+ // Threshold to logging compaction trace, in seconds.
+ CONF_mInt32(base_compaction_trace_threshold, "10");
+ CONF_mInt32(cumulative_compaction_trace_threshold, "2");
Review comment:
Please update the `be-config.md` document, to explain these 2 configs, add how to use the trace.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] acelyc111 commented on pull request #3814: [trace] Adapt trace util to compaction module
Posted by GitBox <gi...@apache.org>.
acelyc111 commented on pull request #3814:
URL: https://github.com/apache/incubator-doris/pull/3814#issuecomment-642776083
> It's a good job. You can add related issue about it.
Added, https://github.com/apache/incubator-doris/issues/3845
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on pull request #3814: [trace] Adapt trace util to compaction module
Posted by GitBox <gi...@apache.org>.
chaoyli commented on pull request #3814:
URL: https://github.com/apache/incubator-doris/pull/3814#issuecomment-642095658
It's a good job. You can add related issue about it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3814: [trace] Adapt trace util to compaction module
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3814:
URL: https://github.com/apache/incubator-doris/pull/3814
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org