You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/02/07 11:30:18 UTC

[PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

HHoflittlefish777 opened a new pull request, #30967:
URL: https://github.com/apache/doris/pull/30967

   ## Proposed changes
   
   #30903
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931931299

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.29 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
   
   query1	0.02	0.02	0.02
   query2	0.06	0.02	0.02
   query3	0.25	0.04	0.04
   query4	1.83	0.06	0.06
   query5	0.53	0.52	0.53
   query6	1.26	0.62	0.62
   query7	0.01	0.01	0.01
   query8	0.04	0.03	0.02
   query9	0.53	0.49	0.47
   query10	0.52	0.54	0.53
   query11	0.11	0.08	0.09
   query12	0.11	0.09	0.09
   query13	0.62	0.61	0.61
   query14	0.78	0.80	0.81
   query15	0.77	0.74	0.76
   query16	0.38	0.36	0.37
   query17	0.98	0.98	1.00
   query18	0.21	0.27	0.27
   query19	1.89	1.85	1.89
   query20	0.01	0.01	0.01
   query21	15.46	0.58	0.57
   query22	2.20	2.47	1.24
   query23	17.24	0.97	0.80
   query24	2.85	0.91	1.36
   query25	1.43	0.12	0.12
   query26	0.21	0.15	0.13
   query27	0.10	0.11	0.11
   query28	9.68	0.73	0.71
   query29	12.68	2.29	2.31
   query30	0.54	0.53	0.50
   query31	2.81	0.38	0.38
   query32	3.42	0.50	0.49
   query33	3.06	3.07	3.05
   query34	15.28	4.80	4.83
   query35	4.88	4.84	4.86
   query36	1.06	1.01	1.02
   query37	0.06	0.05	0.04
   query38	0.04	0.02	0.02
   query39	0.02	0.01	0.02
   query40	0.16	0.14	0.14
   query41	0.06	0.01	0.01
   query42	0.02	0.01	0.01
   query43	0.02	0.01	0.02
   Total cold run time: 104.19 s
   Total hot run time: 30.29 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931936504

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit cc939742641e8856a1185bf653db1f5849205b6f with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #30967:
URL: https://github.com/apache/doris/pull/30967


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931848380

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931922590

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 240715 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
   
   query1	923	386	387	386
   query2	6514	2171	1950	1950
   query3	6914	199	201	199
   query4	20556	18151	18196	18151
   query5	19884	6644	6588	6588
   query6	360	224	260	224
   query7	4694	310	317	310
   query8	273	243	248	243
   query9	3155	2719	2702	2702
   query10	433	289	324	289
   query11	11886	11050	10883	10883
   query12	118	78	74	74
   query13	5762	650	641	641
   query14	20211	13353	13409	13353
   query15	403	233	239	233
   query16	6411	264	268	264
   query17	1664	1460	860	860
   query18	2315	400	405	400
   query19	199	140	143	140
   query20	77	79	74	74
   query21	184	96	90	90
   query22	5402	5187	5116	5116
   query23	32408	31949	31696	31696
   query24	6845	6544	6477	6477
   query25	516	420	415	415
   query26	502	161	158	158
   query27	1791	294	294	294
   query28	6045	2185	2174	2174
   query29	2857	2786	2700	2700
   query30	242	166	163	163
   query31	898	711	723	711
   query32	63	59	60	59
   query33	392	244	233	233
   query34	837	464	476	464
   query35	1132	969	966	966
   query36	1562	1448	1395	1395
   query37	89	58	55	55
   query38	3052	2961	2901	2901
   query39	1374	1311	1324	1311
   query40	202	91	89	89
   query41	35	32	35	32
   query42	82	81	85	81
   query43	578	584	627	584
   query44	1098	712	728	712
   query45	236	226	228	226
   query46	1232	998	961	961
   query47	1804	1637	1714	1637
   query48	962	677	658	658
   query49	607	362	363	362
   query50	860	590	561	561
   query51	5553	5515	5472	5472
   query52	79	92	70	70
   query53	435	316	324	316
   query54	2624	2457	2443	2443
   query55	80	73	80	73
   query56	214	191	205	191
   query57	1170	1082	1011	1011
   query58	208	209	195	195
   query59	3338	3410	3446	3410
   query60	213	185	200	185
   query61	83	85	85	85
   query62	841	562	478	478
   query63	469	328	327	327
   query64	2468	1390	1482	1390
   query65	3647	3549	3560	3549
   query66	750	367	350	350
   query67	15478	15235	15896	15235
   query68	10863	657	679	657
   query69	558	328	341	328
   query70	2568	1547	1689	1547
   query71	425	297	298	297
   query72	6479	3377	3422	3377
   query73	741	331	326	326
   query74	6425	5838	5971	5838
   query75	5634	3720	3635	3635
   query76	6992	1143	1176	1143
   query77	1202	244	238	238
   query78	31876	50854	49735	49735
   query79	14548	649	623	623
   query80	4369	378	384	378
   query81	574	232	228	228
   query82	996	93	92	92
   query83	333	138	126	126
   query84	258	67	67	67
   query85	1849	286	271	271
   query86	476	380	383	380
   query87	3259	3007	2975	2975
   query88	6766	2270	2274	2270
   query89	369	325	271	271
   query90	2500	204	189	189
   query91	162	121	127	121
   query92	62	51	50	50
   query93	4489	557	610	557
   query94	1663	209	210	209
   query95	1137	1067	1070	1067
   query96	644	325	323	323
   query97	6576	6402	6310	6310
   query98	194	185	179	179
   query99	3699	929	883	883
   Total cold run time: 352975 ms
   Total hot run time: 240715 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931848443

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931897857

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 50280 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17677	4381	4340	4340
   q2	2048	145	135	135
   q3	10427	1923	1914	1914
   q4	10382	1209	1276	1209
   q5	8430	3934	3981	3934
   q6	229	126	127	126
   q7	2046	1622	1611	1611
   q8	9518	2704	2724	2704
   q9	13710	10782	10816	10782
   q10	8627	3518	3549	3518
   q11	432	244	245	244
   q12	468	303	298	298
   q13	18345	3943	4045	3943
   q14	344	335	343	335
   q15	495	463	459	459
   q16	706	584	586	584
   q17	1119	976	944	944
   q18	7132	7040	6961	6961
   q19	1658	1536	1487	1487
   q20	519	303	299	299
   q21	4478	4147	4064	4064
   q22	511	403	389	389
   Total cold run time: 119301 ms
   Total hot run time: 50280 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4343	4349	4303	4303
   q2	322	222	228	222
   q3	4208	4172	4105	4105
   q4	2737	2763	2765	2763
   q5	7362	7193	7297	7193
   q6	236	120	123	120
   q7	3269	2826	2808	2808
   q8	4372	4448	4520	4448
   q9	17591	17395	17242	17242
   q10	4258	4270	4241	4241
   q11	750	687	672	672
   q12	1016	878	849	849
   q13	5402	3753	3755	3753
   q14	452	421	442	421
   q15	507	455	455	455
   q16	779	707	714	707
   q17	3877	3823	3899	3823
   q18	8795	8671	8718	8671
   q19	1722	1675	1677	1675
   q20	2433	2192	2150	2150
   q21	8585	8595	8627	8595
   q22	1065	959	976	959
   Total cold run time: 84081 ms
   Total hot run time: 80175 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org