You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/09/25 11:36:23 UTC

[PR] [improvement](binlog) filter dropped indexes #41246 [doris]

w41ter opened a new pull request, #41299:
URL: https://github.com/apache/doris/pull/41299

   cherry pick from #41246


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373836057

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373941159

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 213869 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 9f092c511e3cd8c318011c4211bcbb97df219bb8, data reload: false
   
   query1	928	393	436	393
   query2	6556	2257	2128	2128
   query3	6923	202	208	202
   query4	22874	21797	21455	21455
   query5	19738	6534	6535	6534
   query6	285	221	235	221
   query7	4326	300	305	300
   query8	261	270	251	251
   query9	3060	2637	2582	2582
   query10	468	292	310	292
   query11	15406	15011	14766	14766
   query12	121	76	72	72
   query13	1019	445	435	435
   query14	17488	13709	13349	13349
   query15	387	230	225	225
   query16	6493	291	262	262
   query17	1762	921	930	921
   query18	895	312	317	312
   query19	207	150	151	150
   query20	104	110	107	107
   query21	187	102	93	93
   query22	5198	5025	4967	4967
   query23	34106	33376	33542	33376
   query24	7912	6266	6313	6266
   query25	526	442	431	431
   query26	1272	166	171	166
   query27	2528	296	293	293
   query28	6113	2247	2220	2220
   query29	2924	2808	2803	2803
   query30	240	170	164	164
   query31	955	710	738	710
   query32	68	61	59	59
   query33	450	268	270	268
   query34	859	490	480	480
   query35	1124	920	878	878
   query36	1360	1238	1159	1159
   query37	172	63	60	60
   query38	3088	2891	2898	2891
   query39	1372	1309	1318	1309
   query40	309	95	98	95
   query41	48	38	40	38
   query42	93	99	81	81
   query43	640	570	563	563
   query44	1145	704	710	704
   query45	248	227	232	227
   query46	1235	960	974	960
   query47	2289	1777	1804	1777
   query48	534	411	416	411
   query49	637	380	378	378
   query50	844	612	635	612
   query51	4806	4606	4640	4606
   query52	79	75	73	73
   query53	235	187	190	187
   query54	2660	2510	2518	2510
   query55	81	86	83	83
   query56	237	207	208	207
   query57	1343	1066	1096	1066
   query58	223	215	222	215
   query59	3469	3170	3414	3170
   query60	212	204	191	191
   query61	97	93	95	93
   query62	785	496	456	456
   query63	200	177	173	173
   query64	3571	1594	1479	1479
   query65	3600	3557	3515	3515
   query66	767	411	398	398
   query67	15783	17427	17798	17427
   query68	9104	669	640	640
   query69	477	258	276	258
   query70	1556	1482	1383	1383
   query71	359	312	303	303
   query72	6828	4784	4823	4784
   query73	754	321	331	321
   query74	6262	5812	5816	5812
   query75	4452	3651	3763	3651
   query76	4671	1120	1191	1120
   query77	645	253	247	247
   query78	12564	11548	11800	11548
   query79	8694	654	652	652
   query80	1764	388	392	388
   query81	497	229	236	229
   query82	1699	102	96	96
   query83	166	133	132	132
   query84	261	71	71	71
   query85	990	318	316	316
   query86	329	322	293	293
   query87	3245	3066	3005	3005
   query88	4795	2300	2286	2286
   query89	483	284	299	284
   query90	1846	213	212	212
   query91	158	122	125	122
   query92	60	54	51	51
   query93	6534	539	588	539
   query94	772	215	207	207
   query95	1929	1949	1982	1949
   query96	655	326	321	321
   query97	6466	6448	6358	6358
   query98	231	206	213	206
   query99	2955	840	871	840
   Total cold run time: 318126 ms
   Total hot run time: 213869 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373911062

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 49272 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 9f092c511e3cd8c318011c4211bcbb97df219bb8, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17842	4442	4356	4356
   q2	2090	154	141	141
   q3	10469	1930	1929	1929
   q4	10321	1243	1329	1243
   q5	8619	3884	3928	3884
   q6	236	124	128	124
   q7	2031	1604	1585	1585
   q8	9350	2768	2728	2728
   q9	10222	9902	9801	9801
   q10	8678	3572	3511	3511
   q11	430	252	249	249
   q12	474	302	303	302
   q13	18364	3952	4079	3952
   q14	358	328	323	323
   q15	509	454	457	454
   q16	550	464	450	450
   q17	1154	1001	943	943
   q18	7209	6934	6953	6934
   q19	1705	1597	1544	1544
   q20	542	320	294	294
   q21	4496	4207	4139	4139
   q22	488	386	394	386
   Total cold run time: 116137 ms
   Total hot run time: 49272 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4364	4324	4318	4318
   q2	322	224	219	219
   q3	4170	4131	4130	4130
   q4	2790	2768	2756	2756
   q5	7164	7158	7110	7110
   q6	240	117	122	117
   q7	3220	2846	2826	2826
   q8	4419	4431	4457	4431
   q9	13660	13651	13583	13583
   q10	4248	4248	4259	4248
   q11	748	726	668	668
   q12	1029	855	845	845
   q13	6541	3771	3761	3761
   q14	442	419	436	419
   q15	506	459	456	456
   q16	620	586	564	564
   q17	3811	3842	3852	3842
   q18	8854	8729	8814	8729
   q19	1733	1682	1689	1682
   q20	2391	2103	2116	2103
   q21	8394	8532	8419	8419
   q22	1064	917	971	917
   Total cold run time: 80730 ms
   Total hot run time: 76143 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373852385

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373959305

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 9f092c511e3cd8c318011c4211bcbb97df219bb8 with default session variables
   Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
   Stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       21.0 seconds inserted 10000000 Rows, about 476K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373952524

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.86 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 9f092c511e3cd8c318011c4211bcbb97df219bb8, data reload: false
   
   query1	0.02	0.02	0.02
   query2	0.07	0.03	0.03
   query3	0.26	0.05	0.04
   query4	1.79	0.06	0.06
   query5	0.55	0.52	0.52
   query6	1.24	0.62	0.62
   query7	0.02	0.01	0.01
   query8	0.03	0.02	0.02
   query9	0.52	0.50	0.48
   query10	0.54	0.53	0.54
   query11	0.12	0.09	0.09
   query12	0.12	0.09	0.08
   query13	0.62	0.63	0.61
   query14	0.80	0.78	0.79
   query15	0.79	0.76	0.75
   query16	0.36	0.36	0.37
   query17	1.02	0.98	1.02
   query18	0.23	0.23	0.26
   query19	1.90	1.82	1.86
   query20	0.01	0.01	0.01
   query21	15.47	0.56	0.56
   query22	1.85	1.76	1.83
   query23	17.34	1.01	1.02
   query24	6.48	0.81	1.18
   query25	0.40	0.10	0.04
   query26	0.64	0.15	0.15
   query27	0.04	0.03	0.04
   query28	6.72	0.74	0.78
   query29	12.61	2.40	2.37
   query30	0.54	0.53	0.54
   query31	2.80	0.38	0.37
   query32	3.39	0.50	0.50
   query33	3.09	3.12	3.08
   query34	15.24	4.80	4.79
   query35	4.87	4.84	4.83
   query36	1.06	1.02	1.00
   query37	0.06	0.05	0.04
   query38	0.04	0.02	0.02
   query39	0.02	0.01	0.01
   query40	0.17	0.14	0.14
   query41	0.07	0.01	0.01
   query42	0.02	0.01	0.01
   query43	0.02	0.02	0.02
   Total cold run time: 103.95 s
   Total hot run time: 30.86 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #41299:
URL: https://github.com/apache/doris/pull/41299#issuecomment-2373835956

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](binlog) filter dropped indexes #41246 [doris]

Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter merged PR #41299:
URL: https://github.com/apache/doris/pull/41299


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org