You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/07/04 15:01:13 UTC

[PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

luwei16 opened a new pull request, #37318:
URL: https://github.com/apache/doris/pull/37318

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210019724

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 174571 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
   
   query1	931	374	364	364
   query2	6458	2540	2433	2433
   query3	6641	203	211	203
   query4	28630	17533	17040	17040
   query5	3674	493	498	493
   query6	260	165	183	165
   query7	4590	300	301	300
   query8	325	303	294	294
   query9	8634	2521	2483	2483
   query10	596	307	287	287
   query11	12679	9997	10028	9997
   query12	129	92	85	85
   query13	1655	379	385	379
   query14	10328	7813	7958	7813
   query15	242	188	192	188
   query16	7090	331	320	320
   query17	1767	566	560	560
   query18	1684	285	282	282
   query19	201	161	155	155
   query20	92	82	82	82
   query21	214	129	128	128
   query22	4199	4211	3978	3978
   query23	34037	33461	33594	33461
   query24	11143	2826	2848	2826
   query25	644	421	390	390
   query26	1128	173	166	166
   query27	2421	323	344	323
   query28	7277	2226	2205	2205
   query29	907	662	644	644
   query30	263	165	161	161
   query31	958	772	747	747
   query32	96	53	61	53
   query33	766	326	310	310
   query34	961	507	491	491
   query35	746	647	641	641
   query36	1131	996	999	996
   query37	160	79	86	79
   query38	2975	2823	2780	2780
   query39	888	877	836	836
   query40	219	138	127	127
   query41	55	56	53	53
   query42	111	99	105	99
   query43	602	561	540	540
   query44	1229	727	726	726
   query45	197	162	165	162
   query46	1069	748	749	748
   query47	1878	1779	1801	1779
   query48	367	401	296	296
   query49	859	405	415	405
   query50	754	387	383	383
   query51	6879	6758	6798	6758
   query52	100	98	95	95
   query53	367	286	296	286
   query54	921	454	450	450
   query55	78	72	73	72
   query56	283	265	268	265
   query57	1120	1073	1037	1037
   query58	252	235	255	235
   query59	3591	3610	3340	3340
   query60	304	277	273	273
   query61	100	98	95	95
   query62	627	454	439	439
   query63	319	292	292	292
   query64	8910	2219	1803	1803
   query65	3165	3090	3145	3090
   query66	765	341	332	332
   query67	15749	15147	15020	15020
   query68	8495	558	568	558
   query69	743	452	342	342
   query70	1248	1171	1104	1104
   query71	535	287	270	270
   query72	8851	6287	5539	5539
   query73	1793	324	325	324
   query74	6036	5483	5533	5483
   query75	5190	2679	2662	2662
   query76	4879	991	1009	991
   query77	773	305	293	293
   query78	9514	8970	8837	8837
   query79	9232	521	524	521
   query80	1190	473	489	473
   query81	580	226	225	225
   query82	739	109	110	109
   query83	327	190	163	163
   query84	273	91	85	85
   query85	1422	299	298	298
   query86	397	327	314	314
   query87	3278	3233	3132	3132
   query88	4086	2457	2489	2457
   query89	542	384	382	382
   query90	2051	191	192	191
   query91	132	103	103	103
   query92	65	48	49	48
   query93	6547	530	518	518
   query94	1284	213	210	210
   query95	408	311	329	311
   query96	615	273	269	269
   query97	3200	2995	3061	2995
   query98	219	196	196	196
   query99	1182	844	830	830
   Total cold run time: 304216 ms
   Total hot run time: 174571 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209188513

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #37318:
URL: https://github.com/apache/doris/pull/37318


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209890004

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2209966647

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39638 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17606	4402	4276	4276
   q2	2024	196	193	193
   q3	10491	1184	1160	1160
   q4	10203	794	772	772
   q5	7480	2703	2593	2593
   q6	222	135	133	133
   q7	952	607	622	607
   q8	9238	2062	2111	2062
   q9	8841	6492	6492	6492
   q10	8836	3632	3682	3632
   q11	461	240	242	240
   q12	435	233	224	224
   q13	19070	2961	2975	2961
   q14	265	243	235	235
   q15	528	496	489	489
   q16	502	375	373	373
   q17	964	587	645	587
   q18	7989	7477	7458	7458
   q19	7112	1409	1342	1342
   q20	660	326	324	324
   q21	4892	3149	3225	3149
   q22	381	344	336	336
   Total cold run time: 119152 ms
   Total hot run time: 39638 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4383	4215	4223	4215
   q2	363	265	261	261
   q3	2938	2926	2896	2896
   q4	2057	1661	1754	1661
   q5	5583	5500	5443	5443
   q6	237	132	127	127
   q7	2209	1904	1867	1867
   q8	3297	3404	3438	3404
   q9	8679	8721	8789	8721
   q10	4101	3801	3690	3690
   q11	584	516	499	499
   q12	828	635	650	635
   q13	15967	3130	3126	3126
   q14	309	291	269	269
   q15	541	474	475	474
   q16	472	428	418	418
   q17	1831	1516	1505	1505
   q18	8009	8060	7980	7980
   q19	1775	1481	1594	1481
   q20	2166	1894	1865	1865
   q21	5105	4981	4826	4826
   q22	634	555	583	555
   Total cold run time: 72068 ms
   Total hot run time: 55918 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210975455

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #37318:
URL: https://github.com/apache/doris/pull/37318#discussion_r1665833248


##########
be/test/olap/compaction_task_test.cpp:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock-actions.h>

Review Comment:
   warning: 'gmock/gmock-actions.h' file not found [clang-diagnostic-error]
   ```cpp
   #include <gmock/gmock-actions.h>
            ^
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210043929

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 31.13 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.08	0.04	0.03
   query3	0.22	0.04	0.05
   query4	1.67	0.07	0.07
   query5	0.50	0.49	0.51
   query6	1.14	0.72	0.72
   query7	0.02	0.01	0.02
   query8	0.05	0.04	0.04
   query9	0.55	0.50	0.51
   query10	0.54	0.53	0.57
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.59	0.59	0.58
   query14	0.77	0.78	0.78
   query15	0.85	0.81	0.81
   query16	0.36	0.37	0.35
   query17	1.05	0.99	0.96
   query18	0.25	0.22	0.27
   query19	1.94	1.77	1.82
   query20	0.01	0.01	0.02
   query21	15.44	0.75	0.64
   query22	4.16	7.22	2.38
   query23	18.26	1.42	1.31
   query24	2.12	0.24	0.22
   query25	0.15	0.09	0.09
   query26	0.30	0.21	0.21
   query27	0.45	0.23	0.23
   query28	13.28	1.02	0.99
   query29	12.61	3.31	3.32
   query30	0.25	0.06	0.06
   query31	2.86	0.40	0.40
   query32	3.25	0.47	0.48
   query33	2.91	2.89	2.92
   query34	17.08	4.31	4.32
   query35	4.39	4.44	4.39
   query36	0.66	0.48	0.48
   query37	0.19	0.16	0.15
   query38	0.16	0.14	0.14
   query39	0.05	0.03	0.03
   query40	0.14	0.13	0.12
   query41	0.10	0.05	0.05
   query42	0.06	0.05	0.05
   query43	0.04	0.04	0.04
   Total cold run time: 109.84 s
   Total hot run time: 31.13 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #37318:
URL: https://github.com/apache/doris/pull/37318#discussion_r1665841667


##########
be/src/olap/base_tablet.h:
##########
@@ -42,6 +43,8 @@ struct TabletWithVersion {
     int64_t version;
 };
 
+enum CompactionStage { NOT_SCHEDULED, PENDING, EXECUTING };

Review Comment:
   enum class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37318:
URL: https://github.com/apache/doris/pull/37318#issuecomment-2210975415

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org