You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "LiBinfeng-01 (via GitHub)" <gi...@apache.org> on 2024/01/05 14:23:49 UTC

[PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

LiBinfeng-01 opened a new pull request, #29615:
URL: https://github.com/apache/doris/pull/29615

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 closed pull request #29615: [Feat](PipelineX) add distribute exprs to accelerate pipelinex
URL: https://github.com/apache/doris/pull/29615


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878844497

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.28 seconds
    stream load tsv:          575 seconds loaded 74807831229 Bytes, about 124 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          26.9 seconds inserted 10000000 Rows, about 371K ops/s
    storage size: 17188479936 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878867549

   TeamCity be ut coverage result:
    Function Coverage: 36.62% (8620/23539) 
    Line Coverage: 28.66% (70108/244630)
    Region Coverage: 27.63% (36284/131304)
    Branch Coverage: 24.34% (18537/76170)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/497c2d0c41927dc0cc819aca4ae283bf579cbeaa_497c2d0c41927dc0cc819aca4ae283bf579cbeaa/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878840555

   
   TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
   ```
   TPC-DS sf100 test result on commit 497c2d0c41927dc0cc819aca4ae283bf579cbeaa, data reload: false
   
   run tpcds-sf100 query with default conf and session variables
   query1	944	359	333	333
   query2	6408	2076	1939	1939
   query3	6649	207	203	203
   query4	26606	22475	22400	22400
   query5	3889	510	530	510
   query6	264	184	188	184
   query7	4580	277	259	259
   query8	229	200	192	192
   query9	8297	2575	2602	2575
   query10	424	255	250	250
   query11	16219	15530	15482	15482
   query12	132	76	78	76
   query13	1632	332	317	317
   query14	11397	7217	7186	7186
   query15	216	182	189	182
   query16	6467	280	260	260
   query17	1868	494	497	494
   query18	1962	286	260	260
   query19	195	136	129	129
   query20	82	74	76	74
   query21	198	98	95	95
   query22	4863	4589	4407	4407
   query23	31944	31449	31207	31207
   query24	12094	2815	2812	2812
   query25	571	342	332	332
   query26	1737	143	146	143
   query27	2882	270	277	270
   query28	7103	1909	1922	1909
   query29	2058	372	383	372
   query30	286	147	145	145
   query31	978	755	758	755
   query32	91	53	56	53
   query33	734	255	270	255
   query34	900	449	437	437
   query35	888	750	793	750
   query36	1304	1243	1193	1193
   query37	185	73	71	71
   query38	3385	3273	3270	3270
   query39	1341	1284	1269	1269
   query40	302	93	83	83
   query41	38	39	35	35
   query42	94	82	83	82
   query43	552	487	523	487
   query44	1048	696	699	696
   query45	196	195	186	186
   query46	1074	636	633	633
   query47	1743	1534	1498	1498
   query48	331	257	255	255
   query49	1223	306	306	306
   query50	803	363	327	327
   query51	5399	5252	5232	5232
   query52	95	79	91	79
   query53	209	157	145	145
   query54	1419	561	565	561
   query55	92	85	88	85
   query56	194	195	195	195
   query57	1066	968	958	958
   query58	225	190	197	190
   query59	2895	2611	2755	2611
   query60	237	217	210	210
   query61	85	84	86	84
   query62	661	460	430	430
   query63	173	145	146	145
   query64	5882	1662	1743	1662
   query65	3334	3279	3270	3270
   query66	1386	349	334	334
   query67	15939	15383	15268	15268
   query68	9752	527	532	527
   query69	501	254	244	244
   query70	1779	1498	1584	1498
   query71	471	207	208	207
   query72	5172	3551	3543	3543
   query73	1914	310	314	310
   query74	7005	6445	6382	6382
   query75	4638	2302	2320	2302
   query76	6112	1031	1090	1031
   query77	661	250	254	250
   query78	9167	8683	8707	8683
   query79	1017	507	522	507
   query80	735	355	358	355
   query81	452	214	211	211
   query82	228	94	96	94
   query83	192	140	138	138
   query84	243	54	52	52
   query85	925	280	270	270
   query86	387	411	406	406
   query87	3546	3382	3352	3352
   query88	2855	2258	2247	2247
   query89	334	250	249	249
   query90	1893	199	189	189
   query91	116	89	93	89
   query92	57	52	50	50
   query93	1048	500	439	439
   query94	778	184	188	184
   query95	465	413	415	413
   query96	614	317	316	316
   query97	4316	4184	4164	4164
   query98	208	200	201	200
   query99	1118	787	831	787
   Total cold run time: 287396 ms
   Total hot run time: 178786 ms
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878824653

   
   TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
   ```
   Tpch sf100 test result on commit 497c2d0c41927dc0cc819aca4ae283bf579cbeaa, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18824	5293	5207	5207
   q2	2039	156	144	144
   q3	10609	1130	1143	1130
   q4	10225	780	824	780
   q5	7824	3051	2969	2969
   q6	215	131	131	131
   q7	945	525	545	525
   q8	9251	2070	2004	2004
   q9	6933	6465	6425	6425
   q10	8263	3104	3014	3014
   q11	420	222	209	209
   q12	390	230	227	227
   q13	18031	3691	3637	3637
   q14	254	211	221	211
   q15	559	524	516	516
   q16	464	393	393	393
   q17	989	491	464	464
   q18	7349	6747	6691	6691
   q19	1607	1381	1362	1362
   q20	720	344	344	344
   q21	2857	2378	2434	2378
   q22	361	310	328	310
   Total cold run time: 109129 ms
   Total hot run time: 39071 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5090	5122	5086	5086
   q2	343	235	252	235
   q3	3334	3313	3303	3303
   q4	2132	2021	2028	2021
   q5	5848	5823	5841	5823
   q6	215	121	126	121
   q7	2327	1908	1940	1908
   q8	3429	3480	3491	3480
   q9	8886	8919	8938	8919
   q10	3813	3847	3881	3847
   q11	615	466	494	466
   q12	807	663	639	639
   q13	7041	3208	3181	3181
   q14	298	269	270	269
   q15	577	524	519	519
   q16	580	502	504	502
   q17	1948	1810	1767	1767
   q18	8766	8384	8279	8279
   q19	1648	1608	1633	1608
   q20	2176	1986	1966	1966
   q21	5597	5430	5335	5335
   q22	537	510	485	485
   Total cold run time: 66007 ms
   Total hot run time: 59759 ms
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878772285

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Feat](PipelineX) add distribute exprs to accelerate pipelinex [doris]

Posted by "LiBinfeng-01 (via GitHub)" <gi...@apache.org>.
LiBinfeng-01 commented on PR #29615:
URL: https://github.com/apache/doris/pull/29615#issuecomment-1878761942

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org