You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2024/03/28 02:49:24 UTC

[PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner' default value to 16384 [doris]

mrhhsg opened a new pull request, #32939:
URL: https://github.com/apache/doris/pull/32939

   ## Proposed changes
   
   The previous default value (2M) is too large, which results in reduced parallel performance.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304473

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner' default value to 16384 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024303371

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024346968

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024332840

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38914 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17642	4338	4293	4293
   q2	2082	203	196	196
   q3	10439	1130	1189	1130
   q4	10477	816	770	770
   q5	7577	2945	2833	2833
   q6	214	135	130	130
   q7	1030	623	599	599
   q8	9255	2066	2057	2057
   q9	7790	6908	6825	6825
   q10	8570	3534	3518	3518
   q11	466	239	235	235
   q12	446	216	215	215
   q13	18911	2938	2971	2938
   q14	286	230	243	230
   q15	518	480	482	480
   q16	527	413	397	397
   q17	970	555	629	555
   q18	7422	6888	6721	6721
   q19	4679	1451	1468	1451
   q20	651	308	304	304
   q21	3467	2723	2947	2723
   q22	377	314	328	314
   Total cold run time: 113796 ms
   Total hot run time: 38914 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4347	4128	4182	4128
   q2	379	263	270	263
   q3	2973	2697	2756	2697
   q4	1875	1481	1511	1481
   q5	5392	5352	5422	5352
   q6	208	121	124	121
   q7	2298	1957	1968	1957
   q8	3232	3324	3323	3323
   q9	8875	8861	8882	8861
   q10	3870	3692	3604	3604
   q11	585	498	501	498
   q12	756	613	593	593
   q13	17719	3005	3038	3005
   q14	301	271	273	271
   q15	512	484	467	467
   q16	500	456	455	455
   q17	1773	1428	1453	1428
   q18	7644	7488	7346	7346
   q19	1644	1516	1534	1516
   q20	1951	1787	1770	1770
   q21	5136	4829	4806	4806
   q22	549	469	460	460
   Total cold run time: 72519 ms
   Total hot run time: 54402 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024341048

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 182629 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb, data reload: false
   
   query1	913	396	374	374
   query2	6482	2029	1881	1881
   query3	6672	210	210	210
   query4	23933	21384	21317	21317
   query5	4303	430	421	421
   query6	267	176	193	176
   query7	4613	300	291	291
   query8	236	184	177	177
   query9	8402	2442	2429	2429
   query10	532	248	258	248
   query11	14792	14256	14353	14256
   query12	144	94	93	93
   query13	1640	375	365	365
   query14	10003	7256	7618	7256
   query15	246	209	194	194
   query16	8008	271	261	261
   query17	1927	562	540	540
   query18	2027	283	278	278
   query19	203	152	153	152
   query20	95	88	86	86
   query21	205	132	128	128
   query22	5038	4828	4837	4828
   query23	33403	32873	32915	32873
   query24	11737	2838	2854	2838
   query25	625	366	389	366
   query26	1828	159	158	158
   query27	3132	367	360	360
   query28	7465	1907	1896	1896
   query29	1084	658	634	634
   query30	310	160	159	159
   query31	996	729	741	729
   query32	96	61	55	55
   query33	774	271	260	260
   query34	1144	491	479	479
   query35	864	630	639	630
   query36	1050	925	915	915
   query37	292	76	77	76
   query38	3604	3447	3425	3425
   query39	1519	1466	1444	1444
   query40	292	137	134	134
   query41	52	48	46	46
   query42	107	97	96	96
   query43	495	457	451	451
   query44	1229	727	733	727
   query45	275	281	265	265
   query46	1105	716	715	715
   query47	1925	1892	1847	1847
   query48	389	303	310	303
   query49	1192	384	382	382
   query50	780	397	391	391
   query51	6764	6534	6497	6497
   query52	111	94	95	94
   query53	351	282	289	282
   query54	332	253	245	245
   query55	85	79	82	79
   query56	255	235	233	233
   query57	1230	1172	1142	1142
   query58	248	219	219	219
   query59	2826	2605	2598	2598
   query60	293	257	282	257
   query61	114	115	111	111
   query62	655	451	436	436
   query63	313	283	287	283
   query64	6880	4196	4167	4167
   query65	3175	3085	3043	3043
   query66	1445	377	373	373
   query67	15635	14980	15162	14980
   query68	8211	548	540	540
   query69	563	327	328	327
   query70	1221	1177	1195	1177
   query71	518	293	272	272
   query72	6777	2865	2612	2612
   query73	743	330	325	325
   query74	6862	6436	6445	6436
   query75	3883	2274	2308	2274
   query76	4977	912	919	912
   query77	634	282	275	275
   query78	11082	10231	10154	10154
   query79	12711	528	529	528
   query80	1849	456	437	437
   query81	511	238	228	228
   query82	745	90	93	90
   query83	211	170	171	170
   query84	262	84	86	84
   query85	1410	274	263	263
   query86	399	290	272	272
   query87	3746	3590	3563	3563
   query88	5200	2469	2456	2456
   query89	526	379	381	379
   query90	1982	184	182	182
   query91	131	112	104	104
   query92	62	50	48	48
   query93	7373	528	497	497
   query94	1248	188	184	184
   query95	441	347	335	335
   query96	609	264	277	264
   query97	2664	2515	2482	2482
   query98	232	219	217	217
   query99	1293	888	908	888
   Total cold run time: 308233 ms
   Total hot run time: 182629 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32939:
URL: https://github.com/apache/doris/pull/32939


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304455

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]

Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304048

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org