You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HappenLee (via GitHub)" <gi...@apache.org> on 2024/01/11 03:39:26 UTC

[PR] [Session](rf) Change the default min size of bf runtime filter [doris]

HappenLee opened a new pull request, #29837:
URL: https://github.com/apache/doris/pull/29837

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886397255

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38518 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d8af6ea6fb8fed31ac872cf84d207da2fc729b8a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18019	5017	4923	4923
   q2	2037	136	129	129
   q3	10691	1141	1192	1141
   q4	10256	754	810	754
   q5	7775	3161	3151	3151
   q6	198	123	123	123
   q7	860	514	489	489
   q8	9281	1949	2003	1949
   q9	7327	6417	6440	6417
   q10	8250	3095	3107	3095
   q11	411	213	212	212
   q12	364	189	192	189
   q13	18101	3417	3401	3401
   q14	240	216	208	208
   q15	555	518	510	510
   q16	433	377	387	377
   q17	932	483	563	483
   q18	7245	6673	6626	6626
   q19	1584	1361	1354	1354
   q20	586	305	301	301
   q21	2772	2395	2414	2395
   q22	357	303	291	291
   Total cold run time: 108274 ms
   Total hot run time: 38518 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4968	4968	4950	4950
   q2	308	202	195	195
   q3	3316	3300	3285	3285
   q4	2239	2255	2241	2241
   q5	5785	5754	5774	5754
   q6	195	116	117	116
   q7	2307	1834	1840	1834
   q8	3445	3517	3571	3517
   q9	8823	8806	8783	8783
   q10	3738	3821	3819	3819
   q11	550	438	426	426
   q12	778	596	603	596
   q13	7280	3216	3192	3192
   q14	277	266	273	266
   q15	558	518	497	497
   q16	514	470	475	470
   q17	2049	2047	2058	2047
   q18	8813	8368	8262	8262
   q19	1590	1622	1616	1616
   q20	2144	1923	1946	1923
   q21	6019	5517	5767	5517
   q22	565	444	493	444
   Total cold run time: 66261 ms
   Total hot run time: 59750 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886422758

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit d8af6ea6fb8fed31ac872cf84d207da2fc729b8a with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       12.9 seconds inserted 10000000 Rows, about 775K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886192311

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886192566

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886419261

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.89 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit d8af6ea6fb8fed31ac872cf84d207da2fc729b8a, data reload: false
   
   query1	0.07	0.05	0.05
   query2	0.06	0.03	0.02
   query3	0.25	0.11	0.12
   query4	1.76	0.13	0.11
   query5	0.52	0.52	0.52
   query6	1.39	0.64	0.62
   query7	0.01	0.01	0.02
   query8	0.04	0.02	0.02
   query9	0.55	0.50	0.50
   query10	0.55	0.55	0.55
   query11	0.13	0.10	0.09
   query12	0.11	0.10	0.09
   query13	0.61	0.60	0.60
   query14	0.78	0.78	0.82
   query15	0.83	0.80	0.80
   query16	0.36	0.37	0.34
   query17	0.98	0.97	0.97
   query18	0.24	0.26	0.23
   query19	1.80	1.81	1.81
   query20	0.01	0.00	0.00
   query21	15.40	0.58	0.58
   query22	2.45	2.58	2.26
   query23	17.76	0.78	0.79
   query24	16.37	0.59	0.58
   query25	2.18	0.15	0.14
   query26	0.14	0.14	0.13
   query27	0.15	0.15	0.15
   query28	6.69	0.84	0.81
   query29	12.54	3.14	3.16
   query30	0.59	0.47	0.47
   query31	2.78	0.35	0.36
   query32	3.39	0.48	0.49
   query33	3.21	3.20	3.20
   query34	15.86	4.16	4.17
   query35	4.15	4.16	4.16
   query36	1.10	1.04	1.03
   query37	0.07	0.05	0.05
   query38	0.03	0.02	0.02
   query39	0.02	0.02	0.02
   query40	0.16	0.12	0.13
   query41	0.08	0.01	0.01
   query42	0.02	0.02	0.01
   query43	0.02	0.02	0.02
   Total cold run time: 116.21 s
   Total hot run time: 30.89 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #29837:
URL: https://github.com/apache/doris/pull/29837


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886406151

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.55 seconds
    stream load tsv:          558 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          21.1 seconds inserted 10000000 Rows, about 473K ops/s
    storage size: 17183685903 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886345688

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Session](rf) Change the default min size of bf runtime filter to 2048 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29837:
URL: https://github.com/apache/doris/pull/29837#issuecomment-1886408998

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 179567 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d8af6ea6fb8fed31ac872cf84d207da2fc729b8a, data reload: false
   
   query1	938	338	333	333
   query2	6717	1889	1970	1889
   query3	6712	213	205	205
   query4	27312	22266	22176	22176
   query5	4733	499	500	499
   query6	263	184	192	184
   query7	4615	286	276	276
   query8	221	198	205	198
   query9	9195	2706	2718	2706
   query10	661	246	236	236
   query11	16299	15415	15416	15415
   query12	126	74	76	74
   query13	1705	385	413	385
   query14	11505	7358	7175	7175
   query15	235	198	194	194
   query16	6553	251	243	243
   query17	1912	484	479	479
   query18	1954	275	259	259
   query19	289	151	145	145
   query20	84	75	77	75
   query21	200	135	126	126
   query22	4886	4741	4820	4741
   query23	31856	30935	30969	30935
   query24	12221	2842	2813	2813
   query25	552	331	335	331
   query26	1749	157	157	157
   query27	2955	278	285	278
   query28	7048	1859	1866	1859
   query29	2072	711	668	668
   query30	282	135	141	135
   query31	1008	765	784	765
   query32	89	67	59	59
   query33	714	257	254	254
   query34	1074	467	480	467
   query35	929	744	746	744
   query36	1425	1241	1233	1233
   query37	184	68	68	68
   query38	3398	3318	3278	3278
   query39	1338	1299	1279	1279
   query40	339	101	89	89
   query41	41	35	35	35
   query42	102	98	97	97
   query43	543	484	504	484
   query44	1094	713	727	713
   query45	219	183	185	183
   query46	1062	682	680	680
   query47	1547	1556	1592	1556
   query48	392	333	322	322
   query49	1283	316	317	316
   query50	707	320	319	319
   query51	5286	5261	5213	5213
   query52	101	89	93	89
   query53	361	278	299	278
   query54	2087	1598	1622	1598
   query55	97	90	90	90
   query56	222	200	199	199
   query57	1015	935	969	935
   query58	218	195	200	195
   query59	2732	2530	2471	2471
   query60	239	214	212	212
   query61	83	81	82	81
   query62	596	392	406	392
   query63	303	292	295	292
   query64	5849	1656	1654	1654
   query65	3361	3267	3282	3267
   query66	1393	340	332	332
   query67	15471	14990	15059	14990
   query68	12606	550	537	537
   query69	659	386	384	384
   query70	1707	1545	1594	1545
   query71	500	235	248	235
   query72	4924	2814	2859	2814
   query73	2990	324	319	319
   query74	7086	6466	6555	6466
   query75	5316	2325	2272	2272
   query76	6376	866	1120	866
   query77	668	275	284	275
   query78	9914	8538	8700	8538
   query79	1047	527	504	504
   query80	530	351	354	351
   query81	452	207	211	207
   query82	207	95	86	86
   query83	178	120	123	120
   query84	276	73	76	73
   query85	1061	338	337	337
   query86	400	390	385	385
   query87	3591	3420	3362	3362
   query88	3239	2248	2245	2245
   query89	470	378	398	378
   query90	1900	209	201	201
   query91	157	125	127	125
   query92	64	58	54	54
   query93	1386	439	427	427
   query94	1296	185	188	185
   query95	519	490	472	472
   query96	619	324	326	324
   query97	4283	4132	4175	4132
   query98	210	190	188	188
   query99	1000	742	754	742
   Total cold run time: 298383 ms
   Total hot run time: 179567 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org