You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2024/06/04 11:59:23 UTC

[PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

xzj7019 opened a new pull request, #35795:
URL: https://github.com/apache/doris/pull/35795

   ## Proposed changes
   
   Issue Number: close #xxx
   
   Based on real testing perf. data, the doris_scanner_thread_pool_thread_num's setting will have big influence on multi-stream perf. scenarios. Current 2 * num_cores configuration will be too big on the high configuration cluster be, e.g, per-node with 64 cores, under multi-stream testing cases. Perf. data shows that 5 be cluster with 8 streams ad-hoc test cases will have 60% improvement if reduce this configuration from 2 * num_cores to num_cores. 
   This change will be initial adjustment and should have a more complete adaptive strategy in the future.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147434068

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40820 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17642	4353	4275	4275
   q2	2041	198	201	198
   q3	10440	1219	1123	1123
   q4	10224	843	789	789
   q5	7465	2676	2712	2676
   q6	223	134	137	134
   q7	955	600	602	600
   q8	9227	2108	2108	2108
   q9	9327	6650	6675	6650
   q10	9401	3961	3873	3873
   q11	478	254	240	240
   q12	533	241	250	241
   q13	18054	3198	3257	3198
   q14	254	213	221	213
   q15	509	477	485	477
   q16	509	387	392	387
   q17	955	563	714	563
   q18	8316	7731	7737	7731
   q19	7915	1467	1433	1433
   q20	698	327	328	327
   q21	5200	3342	3257	3257
   q22	388	328	327	327
   Total cold run time: 120754 ms
   Total hot run time: 40820 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4632	4441	4461	4441
   q2	374	268	283	268
   q3	3163	2951	2971	2951
   q4	1911	1588	1602	1588
   q5	5480	5545	5457	5457
   q6	231	123	126	123
   q7	2193	1796	1826	1796
   q8	3231	3371	3355	3355
   q9	8610	8600	8693	8600
   q10	4035	3850	3714	3714
   q11	595	489	492	489
   q12	822	618	618	618
   q13	17103	3156	3113	3113
   q14	323	283	265	265
   q15	523	480	480	480
   q16	492	423	444	423
   q17	1796	1476	1509	1476
   q18	8029	7676	7395	7395
   q19	1764	1590	1656	1590
   q20	3025	1781	1774	1774
   q21	7096	4774	4722	4722
   q22	619	569	581	569
   Total cold run time: 76047 ms
   Total hot run time: 55207 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 closed pull request #35795: [opt](scanner) reduce local scanner thread pool thread num
URL: https://github.com/apache/doris/pull/35795


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147366774

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144474754

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144474681

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 closed pull request #35795: [opt](scanner) reduce default scanner thread pool thread num
URL: https://github.com/apache/doris/pull/35795


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147467208

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.77 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
   
   query1	0.03	0.03	0.03
   query2	0.08	0.03	0.03
   query3	0.23	0.06	0.05
   query4	1.69	0.10	0.10
   query5	0.50	0.47	0.49
   query6	1.12	0.72	0.73
   query7	0.02	0.01	0.02
   query8	0.05	0.04	0.04
   query9	0.54	0.48	0.47
   query10	0.54	0.54	0.54
   query11	0.16	0.11	0.12
   query12	0.14	0.12	0.11
   query13	0.60	0.59	0.60
   query14	0.78	0.77	0.79
   query15	0.83	0.80	0.81
   query16	0.36	0.35	0.36
   query17	1.00	1.01	1.02
   query18	0.24	0.25	0.24
   query19	1.76	1.74	1.71
   query20	0.02	0.01	0.01
   query21	15.72	0.66	0.66
   query22	4.10	7.35	2.08
   query23	18.28	1.34	1.22
   query24	2.00	0.24	0.22
   query25	0.14	0.08	0.09
   query26	0.28	0.18	0.18
   query27	0.07	0.07	0.08
   query28	13.18	1.02	0.99
   query29	13.22	3.39	3.29
   query30	0.24	0.06	0.05
   query31	2.86	0.39	0.38
   query32	3.30	0.46	0.46
   query33	2.90	2.90	2.90
   query34	17.18	4.49	4.45
   query35	4.49	4.51	4.59
   query36	0.71	0.48	0.48
   query37	0.18	0.15	0.15
   query38	0.15	0.15	0.15
   query39	0.05	0.03	0.04
   query40	0.16	0.14	0.14
   query41	0.09	0.05	0.04
   query42	0.05	0.04	0.04
   query43	0.04	0.04	0.04
   Total cold run time: 110.08 s
   Total hot run time: 30.77 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147456032

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171693 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
   
   query1	922	378	370	370
   query2	6460	2460	2381	2381
   query3	6645	210	206	206
   query4	18815	17470	17021	17021
   query5	4160	452	465	452
   query6	253	156	162	156
   query7	4594	309	308	308
   query8	337	275	276	275
   query9	8843	2433	2409	2409
   query10	473	298	272	272
   query11	10494	10147	9940	9940
   query12	131	87	84	84
   query13	1651	366	372	366
   query14	10067	6977	6321	6321
   query15	219	181	186	181
   query16	7153	274	263	263
   query17	1006	535	508	508
   query18	1869	278	274	274
   query19	204	160	155	155
   query20	99	82	85	82
   query21	207	134	134	134
   query22	4149	4017	3920	3920
   query23	33593	32960	33163	32960
   query24	11348	2861	2807	2807
   query25	626	364	355	355
   query26	1192	160	185	160
   query27	3015	312	318	312
   query28	7359	2084	2067	2067
   query29	986	636	613	613
   query30	295	148	150	148
   query31	926	769	731	731
   query32	97	54	54	54
   query33	758	288	287	287
   query34	958	494	490	490
   query35	748	626	633	626
   query36	1097	932	903	903
   query37	153	71	70	70
   query38	2875	2711	2732	2711
   query39	883	809	807	807
   query40	208	127	126	126
   query41	56	51	51	51
   query42	122	95	98	95
   query43	593	555	562	555
   query44	1253	736	748	736
   query45	192	169	160	160
   query46	1066	730	762	730
   query47	1872	1782	1746	1746
   query48	363	297	294	294
   query49	995	402	413	402
   query50	779	394	395	394
   query51	6803	6802	6779	6779
   query52	104	94	92	92
   query53	356	290	293	290
   query54	868	455	473	455
   query55	75	72	74	72
   query56	284	277	265	265
   query57	1146	1070	1037	1037
   query58	251	242	264	242
   query59	3580	3094	3372	3094
   query60	296	276	282	276
   query61	90	91	88	88
   query62	654	445	464	445
   query63	315	293	288	288
   query64	9133	2219	1738	1738
   query65	3178	3092	3121	3092
   query66	960	317	328	317
   query67	15327	14687	14819	14687
   query68	4492	553	550	550
   query69	537	382	337	337
   query70	1106	1124	1132	1124
   query71	386	291	285	285
   query72	7171	5599	5946	5599
   query73	733	323	327	323
   query74	5899	5525	5490	5490
   query75	3356	2687	2674	2674
   query76	2412	995	1014	995
   query77	696	307	311	307
   query78	10291	9681	9673	9673
   query79	1950	516	514	514
   query80	919	464	465	464
   query81	554	225	217	217
   query82	1034	109	102	102
   query83	239	171	168	168
   query84	272	84	86	84
   query85	1191	268	279	268
   query86	435	376	335	335
   query87	3302	3089	3041	3041
   query88	3180	2379	2385	2379
   query89	469	392	373	373
   query90	1785	194	190	190
   query91	125	98	96	96
   query92	67	54	52	52
   query93	1808	524	508	508
   query94	1220	194	191	191
   query95	408	316	315	315
   query96	588	274	273	273
   query97	3143	3029	2995	2995
   query98	250	220	212	212
   query99	1294	855	834	834
   Total cold run time: 267891 ms
   Total hot run time: 171693 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147376387

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147415521

   TeamCity be ut coverage result:
    Function Coverage: 37.40% (9283/24823) 
    Line Coverage: 28.73% (76014/264593)
    Region Coverage: 28.12% (39388/140086)
    Branch Coverage: 24.53% (19923/81224)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/a7bf8bebd0abe15ba5726cb138f02d494bd725cd_a7bf8bebd0abe15ba5726cb138f02d494bd725cd/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144469279

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144480920

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org