You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/01/14 14:59:48 UTC

[PR] [fix](multi-table-load) fix multi table load can not finish [doris]

HHoflittlefish777 opened a new pull request, #29957:
URL: https://github.com/apache/doris/pull/29957

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890978037

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893474743

   TeamCity be ut coverage result:
    Function Coverage: 36.67% (8648/23581) 
    Line Coverage: 28.73% (70619/245773)
    Region Coverage: 27.62% (36436/131925)
    Branch Coverage: 24.34% (18640/76590)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/e8b53fbf394389ba99924c201800c50b6abfec85_e8b53fbf394389ba99924c201800c50b6abfec85/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893793160

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29957:
URL: https://github.com/apache/doris/pull/29957


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890987371

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893436689

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890977359

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890990424

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40100 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17947	6301	6020	6020
   q2	2052	152	151	151
   q3	10865	1112	1174	1112
   q4	10522	809	762	762
   q5	7816	3172	3164	3164
   q6	206	124	124	124
   q7	872	515	477	477
   q8	9262	1989	2002	1989
   q9	7299	6453	6418	6418
   q10	8218	3043	3078	3043
   q11	418	205	212	205
   q12	363	193	191	191
   q13	18100	3389	3435	3389
   q14	250	221	210	210
   q15	550	524	533	524
   q16	432	393	384	384
   q17	948	518	542	518
   q18	7563	6988	7022	6988
   q19	1618	1378	1402	1378
   q20	564	307	280	280
   q21	2892	2476	2460	2460
   q22	395	330	313	313
   Total cold run time: 109152 ms
   Total hot run time: 40100 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5727	5582	5814	5582
   q2	348	220	219	219
   q3	3372	3161	3067	3067
   q4	2222	2136	2031	2031
   q5	6037	5933	5984	5933
   q6	204	122	124	122
   q7	2356	1942	2013	1942
   q8	3657	3694	3790	3694
   q9	9685	26064	9551	9551
   q10	4145	3780	3841	3780
   q11	595	449	486	449
   q12	1168	650	640	640
   q13	17107	3294	3247	3247
   q14	310	267	275	267
   q15	565	508	519	508
   q16	544	475	472	472
   q17	2515	2040	1994	1994
   q18	9641	9475	12300	9475
   q19	12792	1624	1634	1624
   q20	3831	1960	1938	1938
   q21	14733	5845	5602	5602
   q22	980	635	594	594
   Total cold run time: 102534 ms
   Total hot run time: 62731 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891974119

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893744010

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893859515

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38951 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17716	5530	5300	5300
   q2	2036	142	137	137
   q3	10733	1169	1159	1159
   q4	10286	789	727	727
   q5	7766	3191	3196	3191
   q6	197	123	120	120
   q7	846	487	499	487
   q8	9223	1971	1980	1971
   q9	7337	6413	6389	6389
   q10	8235	3043	3079	3043
   q11	433	212	193	193
   q12	356	190	187	187
   q13	18015	3378	3329	3329
   q14	242	208	216	208
   q15	539	503	510	503
   q16	467	373	370	370
   q17	955	515	476	476
   q18	7405	6911	6826	6826
   q19	1827	1354	1408	1354
   q20	556	309	283	283
   q21	2832	2400	2446	2400
   q22	348	303	298	298
   Total cold run time: 108350 ms
   Total hot run time: 38951 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5443	5339	5203	5203
   q2	327	213	218	213
   q3	3347	3216	3193	3193
   q4	2089	2057	2031	2031
   q5	6017	5929	5798	5798
   q6	197	116	113	113
   q7	2315	1849	1875	1849
   q8	3222	3360	3398	3360
   q9	8947	8882	8837	8837
   q10	3937	3789	3775	3775
   q11	554	457	441	441
   q12	794	652	634	634
   q13	16900	3149	3136	3136
   q14	298	256	260	256
   q15	548	504	505	504
   q16	548	480	480	480
   q17	1855	1889	1875	1875
   q18	9558	19567	9377	9377
   q19	24733	1557	1510	1510
   q20	4605	1933	1932	1932
   q21	14402	5498	5391	5391
   q22	973	547	544	544
   Total cold run time: 111609 ms
   Total hot run time: 60452 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890993277

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.66 seconds
    stream load tsv:          567 seconds loaded 74807831229 Bytes, about 125 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          21.1 seconds inserted 10000000 Rows, about 473K ops/s
    storage size: 17183923051 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453089406


##########
be/src/io/fs/multi_table_pipe.h:
##########
@@ -46,7 +46,13 @@ class MultiTablePipe : public KafkaConsumerPipe {
     // request and execute plans for unplanned pipes
     Status request_and_exec_plans();
 
-    void set_consume_finished() { _consume_finished.store(true, std::memory_order_release); }
+    void set_consume_finished() {
+        --_inflight_cnt;

Review Comment:
   why need do `--_inflight_cnt` here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893492763

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 177369 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
   
   query1	938	329	336	329
   query2	6544	2074	1928	1928
   query3	6711	199	197	197
   query4	33718	22282	22172	22172
   query5	6928	601	600	600
   query6	264	177	174	174
   query7	4607	265	263	263
   query8	226	176	186	176
   query9	8431	2507	2476	2476
   query10	421	234	226	226
   query11	16733	15341	15268	15268
   query12	120	66	65	65
   query13	1689	365	383	365
   query14	10499	6828	6870	6828
   query15	211	182	181	181
   query16	4602	241	238	238
   query17	980	484	477	477
   query18	1789	264	259	259
   query19	174	143	133	133
   query20	71	70	66	66
   query21	192	137	131	131
   query22	5100	4859	4852	4852
   query23	31517	30764	30677	30677
   query24	11443	2783	2743	2743
   query25	564	317	310	310
   query26	1620	144	143	143
   query27	3214	278	267	267
   query28	7171	1851	1847	1847
   query29	1395	653	684	653
   query30	278	142	147	142
   query31	926	746	763	746
   query32	72	51	48	48
   query33	691	225	215	215
   query34	1115	464	462	462
   query35	900	802	764	764
   query36	1422	1262	1281	1262
   query37	91	58	57	57
   query38	3355	3224	3208	3208
   query39	1310	1288	1255	1255
   query40	206	92	85	85
   query41	38	34	34	34
   query42	89	82	89	82
   query43	540	516	474	474
   query44	1041	690	715	690
   query45	200	181	176	176
   query46	1071	657	655	655
   query47	1703	1504	1553	1504
   query48	400	315	312	312
   query49	1132	294	286	286
   query50	700	310	312	310
   query51	5302	5205	5215	5205
   query52	87	87	78	78
   query53	338	276	281	276
   query54	891	440	441	440
   query55	86	88	86	86
   query56	178	177	181	177
   query57	999	916	938	916
   query58	197	163	174	163
   query59	2866	2734	2729	2729
   query60	227	195	195	195
   query61	84	83	89	83
   query62	679	396	401	396
   query63	307	265	264	264
   query64	5015	1754	1735	1735
   query65	3353	3251	3262	3251
   query66	1244	331	320	320
   query67	15927	15415	15310	15310
   query68	10669	526	539	526
   query69	623	303	300	300
   query70	1782	1610	1603	1603
   query71	548	213	225	213
   query72	4834	2826	2834	2826
   query73	2117	316	315	315
   query74	7538	6454	6383	6383
   query75	4728	2334	2277	2277
   query76	6257	938	1001	938
   query77	664	245	246	245
   query78	9144	9040	8708	8708
   query79	1031	499	495	495
   query80	684	318	318	318
   query81	466	204	212	204
   query82	184	81	79	79
   query83	146	119	118	118
   query84	277	69	71	69
   query85	1087	346	336	336
   query86	384	397	381	381
   query87	3468	3377	3345	3345
   query88	3277	2190	2185	2185
   query89	444	363	369	363
   query90	1998	191	186	186
   query91	149	124	129	124
   query92	56	55	46	46
   query93	3178	434	419	419
   query94	1209	167	162	162
   query95	498	457	454	454
   query96	648	311	307	307
   query97	4240	4148	4138	4138
   query98	205	186	187	186
   query99	989	685	685	685
   Total cold run time: 296454 ms
   Total hot run time: 177369 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893473982

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39614 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18544	5709	5864	5709
   q2	2419	152	141	141
   q3	11946	1131	1200	1131
   q4	10844	840	834	834
   q5	7972	3096	3190	3096
   q6	220	127	131	127
   q7	915	504	485	485
   q8	9215	1942	2010	1942
   q9	7310	6398	6364	6364
   q10	8207	3026	3057	3026
   q11	403	219	205	205
   q12	363	196	198	196
   q13	17998	3425	3386	3386
   q14	252	218	214	214
   q15	555	533	503	503
   q16	445	409	387	387
   q17	956	485	481	481
   q18	7663	6984	6966	6966
   q19	1593	1307	1438	1307
   q20	541	320	345	320
   q21	2926	2484	2532	2484
   q22	386	311	310	310
   Total cold run time: 111673 ms
   Total hot run time: 39614 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5868	5804	5657	5657
   q2	336	223	221	221
   q3	3382	3320	3255	3255
   q4	2165	2099	1995	1995
   q5	6165	5957	6064	5957
   q6	208	123	124	123
   q7	2318	1977	1923	1923
   q8	3320	3505	3511	3505
   q9	9390	9011	8757	8757
   q10	6371	3706	3918	3706
   q11	568	457	446	446
   q12	863	641	665	641
   q13	16958	3238	3199	3199
   q14	291	263	280	263
   q15	550	511	511	511
   q16	544	466	475	466
   q17	2031	1979	1973	1973
   q18	9641	9504	15925	9504
   q19	23379	1699	1634	1634
   q20	4602	1963	1939	1939
   q21	13809	5520	5494	5494
   q22	991	570	532	532
   Total cold run time: 113750 ms
   Total hot run time: 61701 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985269

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 178647 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false
   
   query1	939	337	328	328
   query2	6736	1831	1868	1831
   query3	6712	221	211	211
   query4	34784	22172	22150	22150
   query5	6953	527	585	527
   query6	262	183	186	183
   query7	4608	284	277	277
   query8	233	197	201	197
   query9	9164	2702	2781	2702
   query10	481	243	238	238
   query11	16556	15278	15193	15193
   query12	130	71	70	70
   query13	1690	384	381	381
   query14	11259	7442	7415	7415
   query15	216	184	186	184
   query16	4618	260	255	255
   query17	994	495	473	473
   query18	1803	267	268	267
   query19	180	149	147	147
   query20	80	80	74	74
   query21	189	122	124	122
   query22	5106	4994	4799	4799
   query23	31822	31845	31158	31158
   query24	12709	2823	2780	2780
   query25	569	328	328	328
   query26	1854	152	155	152
   query27	3623	288	275	275
   query28	7485	1845	1840	1840
   query29	2230	696	676	676
   query30	283	138	145	138
   query31	942	756	772	756
   query32	75	63	61	61
   query33	712	243	255	243
   query34	1295	459	456	456
   query35	899	771	757	757
   query36	1394	1256	1260	1256
   query37	186	67	71	67
   query38	3431	3249	3244	3244
   query39	1327	1273	1290	1273
   query40	341	99	96	96
   query41	39	35	35	35
   query42	99	94	102	94
   query43	558	508	531	508
   query44	1109	689	697	689
   query45	204	186	187	186
   query46	1045	655	655	655
   query47	1690	1587	1578	1578
   query48	421	332	310	310
   query49	1246	315	302	302
   query50	703	318	309	309
   query51	5312	5308	5215	5215
   query52	106	86	99	86
   query53	354	298	313	298
   query54	897	449	451	449
   query55	96	93	88	88
   query56	209	196	206	196
   query57	1022	969	934	934
   query58	216	195	193	193
   query59	2854	2574	2595	2574
   query60	237	215	230	215
   query61	86	95	89	89
   query62	697	396	408	396
   query63	299	291	296	291
   query64	6250	1667	1652	1652
   query65	3339	3274	3276	3274
   query66	1422	335	332	332
   query67	15413	15040	14931	14931
   query68	12319	518	501	501
   query69	657	335	338	335
   query70	1702	1616	1536	1536
   query71	584	259	233	233
   query72	5021	2858	2902	2858
   query73	3090	322	313	313
   query74	7549	6407	6473	6407
   query75	5680	2355	2310	2310
   query76	6586	881	1094	881
   query77	984	264	307	264
   query78	9226	8709	8596	8596
   query79	1257	503	509	503
   query80	645	340	351	340
   query81	464	210	208	208
   query82	199	95	88	88
   query83	209	122	121	121
   query84	271	72	73	72
   query85	1108	344	336	336
   query86	383	397	384	384
   query87	3605	3387	3372	3372
   query88	3466	2291	2297	2291
   query89	452	405	395	395
   query90	2250	214	196	196
   query91	153	138	135	135
   query92	62	55	52	52
   query93	1411	407	404	404
   query94	1534	180	177	177
   query95	512	490	472	472
   query96	636	330	331	330
   query97	4263	4168	4194	4168
   query98	194	188	188	188
   query99	1082	722	738	722
   Total cold run time: 308347 ms
   Total hot run time: 178647 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890994759

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.94 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
   
   query1	0.05	0.06	0.05
   query2	0.06	0.03	0.02
   query3	0.26	0.12	0.11
   query4	1.78	0.13	0.11
   query5	0.54	0.53	0.53
   query6	1.32	0.65	0.64
   query7	0.02	0.01	0.01
   query8	0.04	0.02	0.02
   query9	0.55	0.49	0.51
   query10	0.56	0.58	0.57
   query11	0.12	0.09	0.09
   query12	0.12	0.10	0.09
   query13	0.60	0.60	0.60
   query14	0.78	0.78	0.79
   query15	0.80	0.78	0.77
   query16	0.38	0.36	0.35
   query17	1.01	0.98	1.04
   query18	0.24	0.25	0.24
   query19	1.89	1.81	1.78
   query20	0.02	0.01	0.01
   query21	15.41	0.59	0.56
   query22	2.52	1.75	2.12
   query23	17.29	0.77	0.87
   query24	2.43	1.18	0.92
   query25	1.98	0.16	0.15
   query26	0.14	0.13	0.14
   query27	0.14	0.15	0.15
   query28	11.02	0.80	0.80
   query29	12.56	3.17	3.16
   query30	0.49	0.48	0.48
   query31	2.78	0.35	0.35
   query32	3.39	0.48	0.50
   query33	3.22	3.25	3.25
   query34	15.85	4.34	4.21
   query35	4.25	4.23	4.23
   query36	1.12	1.09	1.07
   query37	0.06	0.05	0.05
   query38	0.03	0.04	0.03
   query39	0.02	0.01	0.02
   query40	0.16	0.12	0.13
   query41	0.07	0.01	0.02
   query42	0.02	0.01	0.02
   query43	0.03	0.01	0.02
   Total cold run time: 106.12 s
   Total hot run time: 30.94 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891246975

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893908188

   TeamCity be ut coverage result:
    Function Coverage: 36.67% (8648/23586) 
    Line Coverage: 28.73% (70623/245834)
    Region Coverage: 27.62% (36443/131943)
    Branch Coverage: 24.34% (18644/76598)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/838d55e9adecd7024bcf2b7aec9f4ef6aeb30380_838d55e9adecd7024bcf2b7aec9f4ef6aeb30380/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893807545

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890982261

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39943 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18139	6046	6285	6046
   q2	2051	158	135	135
   q3	10667	1152	1152	1152
   q4	10279	786	822	786
   q5	7823	3216	3176	3176
   q6	203	125	125	125
   q7	900	491	484	484
   q8	9296	1963	2002	1963
   q9	7278	6433	6431	6431
   q10	8242	3048	3099	3048
   q11	422	220	209	209
   q12	369	191	194	191
   q13	18110	3418	3423	3418
   q14	245	230	219	219
   q15	554	527	512	512
   q16	444	412	398	398
   q17	952	503	503	503
   q18	7802	7233	6808	6808
   q19	1637	1329	1560	1329
   q20	569	319	297	297
   q21	2926	2442	2397	2397
   q22	365	333	316	316
   Total cold run time: 109273 ms
   Total hot run time: 39943 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5849	5672	6100	5672
   q2	347	223	232	223
   q3	3369	3122	3141	3122
   q4	2216	2027	2090	2027
   q5	6139	5985	5917	5917
   q6	211	128	123	123
   q7	2433	2023	1968	1968
   q8	3579	3658	3733	3658
   q9	9770	20972	10042	10042
   q10	8615	3658	3865	3658
   q11	593	473	462	462
   q12	1191	628	682	628
   q13	17080	3262	3236	3236
   q14	322	266	257	257
   q15	584	508	509	508
   q16	550	487	485	485
   q17	3689	1966	2069	1966
   q18	9612	9642	10520	9642
   q19	12931	1637	1623	1623
   q20	3950	1953	1940	1940
   q21	13194	5727	5731	5727
   q22	1537	598	622	598
   Total cold run time: 107761 ms
   Total hot run time: 63482 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890990402

   TeamCity be ut coverage result:
    Function Coverage: 36.70% (8649/23569) 
    Line Coverage: 28.71% (70445/245338)
    Region Coverage: 27.63% (36411/131786)
    Branch Coverage: 24.34% (18620/76486)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985089

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.79 seconds
    stream load tsv:          561 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          21.6 seconds inserted 10000000 Rows, about 462K ops/s
    storage size: 17183973713 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985491

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891275866

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.59 seconds
    stream load tsv:          564 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          21.9 seconds inserted 10000000 Rows, about 456K ops/s
    storage size: 17183914922 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893882077

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 176822 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
   
   query1	937	340	328	328
   query2	6564	1943	1842	1842
   query3	6705	208	207	207
   query4	31434	22287	22097	22097
   query5	6905	597	594	594
   query6	264	183	175	175
   query7	4606	262	257	257
   query8	227	178	177	177
   query9	8374	2525	2516	2516
   query10	413	239	220	220
   query11	16666	15512	15593	15512
   query12	117	70	74	70
   query13	1706	377	370	370
   query14	10490	6782	6814	6782
   query15	206	172	176	172
   query16	4611	241	227	227
   query17	981	485	475	475
   query18	1788	259	250	250
   query19	168	138	130	130
   query20	70	69	62	62
   query21	187	127	125	125
   query22	4904	4896	4809	4809
   query23	32006	30812	30883	30812
   query24	11484	2829	2817	2817
   query25	536	312	310	310
   query26	884	134	134	134
   query27	2890	279	266	266
   query28	6361	1802	1808	1802
   query29	925	616	628	616
   query30	284	136	144	136
   query31	945	756	763	756
   query32	69	47	48	47
   query33	701	219	207	207
   query34	1143	452	454	452
   query35	858	762	735	735
   query36	1338	1287	1290	1287
   query37	91	60	58	58
   query38	3428	3260	3222	3222
   query39	1317	1274	1252	1252
   query40	192	86	85	85
   query41	38	35	35	35
   query42	92	83	83	83
   query43	525	500	508	500
   query44	1051	677	693	677
   query45	193	188	179	179
   query46	1065	663	663	663
   query47	1693	1560	1628	1560
   query48	397	303	312	303
   query49	1096	292	278	278
   query50	689	304	303	303
   query51	5312	5206	5172	5172
   query52	83	84	75	75
   query53	321	252	263	252
   query54	887	428	443	428
   query55	79	78	80	78
   query56	170	162	172	162
   query57	1011	933	954	933
   query58	183	159	156	156
   query59	2826	2692	2753	2692
   query60	216	183	180	180
   query61	84	81	93	81
   query62	639	412	395	395
   query63	272	259	252	252
   query64	4545	1801	1766	1766
   query65	3358	3279	3234	3234
   query66	1233	315	306	306
   query67	15810	15146	14911	14911
   query68	12882	519	495	495
   query69	628	300	298	298
   query70	2066	1550	1488	1488
   query71	554	210	202	202
   query72	5422	2814	2847	2814
   query73	3256	319	320	319
   query74	7061	6485	6422	6422
   query75	5225	2354	2297	2297
   query76	6191	1018	1069	1018
   query77	714	243	237	237
   query78	9107	8614	8553	8553
   query79	1615	481	487	481
   query80	536	316	317	316
   query81	468	206	206	206
   query82	192	82	74	74
   query83	168	120	118	118
   query84	271	71	67	67
   query85	1063	318	315	315
   query86	388	410	382	382
   query87	3545	3366	3376	3366
   query88	3023	2261	2256	2256
   query89	438	355	360	355
   query90	2102	188	187	187
   query91	156	123	123	123
   query92	58	45	42	42
   query93	1364	401	427	401
   query94	1236	159	153	153
   query95	491	454	434	434
   query96	613	328	324	324
   query97	4277	4133	4147	4133
   query98	207	185	181	181
   query99	1009	728	720	720
   Total cold run time: 293969 ms
   Total hot run time: 176822 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891260591

   TeamCity be ut coverage result:
    Function Coverage: 36.68% (8648/23577) 
    Line Coverage: 28.70% (70454/245455)
    Region Coverage: 27.63% (36421/131818)
    Branch Coverage: 24.35% (18628/76514)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893892469

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.75 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.06	0.03	0.02
   query3	0.22	0.06	0.05
   query4	1.69	0.07	0.06
   query5	0.54	0.52	0.53
   query6	1.31	0.66	0.64
   query7	0.02	0.01	0.01
   query8	0.04	0.03	0.03
   query9	0.55	0.50	0.49
   query10	0.56	0.56	0.57
   query11	0.11	0.08	0.08
   query12	0.10	0.09	0.09
   query13	0.60	0.61	0.60
   query14	0.76	0.80	0.79
   query15	0.80	0.78	0.76
   query16	0.38	0.38	0.37
   query17	1.02	0.99	1.03
   query18	0.25	0.24	0.25
   query19	1.82	1.75	1.79
   query20	0.01	0.01	0.02
   query21	15.41	0.59	0.59
   query22	2.81	2.38	1.84
   query23	17.31	0.79	0.74
   query24	2.35	1.65	0.90
   query25	0.41	0.22	0.20
   query26	0.63	0.14	0.13
   query27	0.06	0.05	0.05
   query28	11.27	0.75	0.77
   query29	12.52	3.10	3.15
   query30	0.62	0.55	0.50
   query31	2.77	0.33	0.35
   query32	3.39	0.48	0.48
   query33	3.25	3.24	3.24
   query34	15.76	4.27	4.26
   query35	4.24	4.28	4.23
   query36	1.12	1.06	1.07
   query37	0.07	0.05	0.06
   query38	0.04	0.03	0.03
   query39	0.02	0.01	0.01
   query40	0.16	0.13	0.13
   query41	0.06	0.01	0.02
   query42	0.02	0.02	0.01
   query43	0.02	0.02	0.02
   Total cold run time: 105.19 s
   Total hot run time: 30.75 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453293240


##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -261,20 +261,9 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env, std::vector<ExecParam> para
                         _status = *status;
                     }
 
-                    --_inflight_plan_cnt;
-                    if (_inflight_plan_cnt == 0 && is_consume_finished()) {
-                        _ctx->number_total_rows = _number_total_rows;
-                        _ctx->number_loaded_rows = _number_loaded_rows;
-                        _ctx->number_filtered_rows = _number_filtered_rows;
-                        _ctx->number_unselected_rows = _number_unselected_rows;
-                        _ctx->commit_infos = _tablet_commit_infos;
-                        LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
-                                  << _ctx->number_total_rows
-                                  << " number_loaded_rows=" << _ctx->number_loaded_rows
-                                  << " number_filtered_rows=" << _ctx->number_filtered_rows
-                                  << " number_unselected_rows=" << _ctx->number_unselected_rows;
-                        _ctx->promise.set_value(
-                                _status); // when all done, finish the routine load task
+                    --_inflight_cnt;
+                    if (_inflight_cnt == 0 && is_consume_finished()) {

Review Comment:
   use fetch_sub(1) to avoid multi threads competition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1894987538

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890979279

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893501488

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.81 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
   
   query1	0.03	0.03	0.03
   query2	0.06	0.03	0.02
   query3	0.24	0.05	0.05
   query4	1.69	0.06	0.06
   query5	0.53	0.52	0.52
   query6	1.25	0.66	0.63
   query7	0.02	0.01	0.01
   query8	0.04	0.02	0.03
   query9	0.53	0.48	0.51
   query10	0.56	0.57	0.56
   query11	0.13	0.09	0.09
   query12	0.11	0.09	0.10
   query13	0.60	0.60	0.59
   query14	0.77	0.78	0.79
   query15	0.80	0.78	0.78
   query16	0.39	0.36	0.37
   query17	1.01	1.00	1.01
   query18	0.24	0.24	0.26
   query19	1.88	1.71	1.76
   query20	0.02	0.01	0.01
   query21	15.40	0.56	0.56
   query22	2.48	3.12	1.56
   query23	17.22	1.02	1.01
   query24	2.63	1.11	1.38
   query25	0.35	0.28	0.15
   query26	0.60	0.14	0.14
   query27	0.06	0.06	0.05
   query28	10.72	0.76	0.77
   query29	12.52	3.14	3.23
   query30	0.55	0.49	0.47
   query31	2.78	0.34	0.35
   query32	3.38	0.47	0.48
   query33	3.23	3.24	3.21
   query34	16.07	4.29	4.22
   query35	4.26	4.28	4.24
   query36	1.10	1.09	1.08
   query37	0.07	0.05	0.05
   query38	0.04	0.03	0.03
   query39	0.02	0.02	0.02
   query40	0.18	0.13	0.13
   query41	0.07	0.02	0.01
   query42	0.02	0.01	0.02
   query43	0.02	0.02	0.02
   Total cold run time: 104.67 s
   Total hot run time: 30.81 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890993420

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 179479 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
   
   query1	960	340	328	328
   query2	6720	1912	1851	1851
   query3	6708	220	212	212
   query4	32297	22414	22099	22099
   query5	7011	542	531	531
   query6	269	197	196	196
   query7	4635	282	276	276
   query8	239	201	201	201
   query9	9494	2757	2857	2757
   query10	453	242	229	229
   query11	16574	15348	15268	15268
   query12	128	75	70	70
   query13	1689	398	394	394
   query14	11239	7387	7418	7387
   query15	229	178	185	178
   query16	4637	251	245	245
   query17	999	500	479	479
   query18	1808	269	265	265
   query19	179	150	152	150
   query20	81	80	75	75
   query21	197	142	135	135
   query22	4907	4811	4738	4738
   query23	32270	31178	31084	31084
   query24	12720	2914	2882	2882
   query25	582	331	329	329
   query26	1835	153	162	153
   query27	3309	279	280	279
   query28	7283	1867	1848	1848
   query29	2223	692	684	684
   query30	284	137	146	137
   query31	941	751	762	751
   query32	76	58	60	58
   query33	717	241	261	241
   query34	1275	470	467	467
   query35	901	817	767	767
   query36	1420	1308	1307	1307
   query37	186	74	73	73
   query38	3425	3291	3285	3285
   query39	1337	1274	1274	1274
   query40	345	102	99	99
   query41	40	38	36	36
   query42	101	102	91	91
   query43	544	535	509	509
   query44	1099	699	703	699
   query45	205	190	181	181
   query46	1072	634	652	634
   query47	1623	1605	1538	1538
   query48	401	315	332	315
   query49	1253	307	307	307
   query50	705	317	316	316
   query51	5307	5244	5216	5216
   query52	108	92	95	92
   query53	355	292	295	292
   query54	891	460	470	460
   query55	95	90	90	90
   query56	213	213	197	197
   query57	1024	923	903	903
   query58	218	198	200	198
   query59	2991	2569	2691	2569
   query60	234	216	221	216
   query61	86	96	86	86
   query62	682	423	406	406
   query63	308	283	295	283
   query64	6284	1680	1676	1676
   query65	3359	3269	3257	3257
   query66	1431	352	335	335
   query67	15813	15343	15275	15275
   query68	9651	550	534	534
   query69	659	338	344	338
   query70	1708	1604	1544	1544
   query71	541	252	255	252
   query72	4305	2867	2847	2847
   query73	2053	323	332	323
   query74	7229	6485	6466	6466
   query75	4518	2344	2354	2344
   query76	5782	1012	1068	1012
   query77	857	279	271	271
   query78	9082	8924	8621	8621
   query79	1058	513	496	496
   query80	737	350	361	350
   query81	464	207	203	203
   query82	307	102	91	91
   query83	175	125	133	125
   query84	279	80	73	73
   query85	1138	341	329	329
   query86	388	382	391	382
   query87	3504	3381	3386	3381
   query88	3564	2313	2294	2294
   query89	464	385	397	385
   query90	2069	210	202	202
   query91	162	122	138	122
   query92	64	54	55	54
   query93	1618	412	426	412
   query94	1282	182	182	182
   query95	515	488	473	473
   query96	634	331	331	331
   query97	4287	4156	4262	4156
   query98	206	183	183	183
   query99	1027	756	744	744
   Total cold run time: 299351 ms
   Total hot run time: 179479 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893449446

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893898302

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453089406


##########
be/src/io/fs/multi_table_pipe.h:
##########
@@ -46,7 +46,13 @@ class MultiTablePipe : public KafkaConsumerPipe {
     // request and execute plans for unplanned pipes
     Status request_and_exec_plans();
 
-    void set_consume_finished() { _consume_finished.store(true, std::memory_order_release); }
+    void set_consume_finished() {
+        --_inflight_cnt;

Review Comment:
   why need do `--_inflight_cnt` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893757891

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893506230

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit e8b53fbf394389ba99924c201800c50b6abfec85 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890975923

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891264947

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40008 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18053	6242	5932	5932
   q2	2041	150	132	132
   q3	10910	1193	1165	1165
   q4	10544	809	802	802
   q5	7829	3159	3178	3159
   q6	204	124	127	124
   q7	889	518	489	489
   q8	9294	2009	1998	1998
   q9	7270	6444	6388	6388
   q10	8233	3090	3089	3089
   q11	416	207	222	207
   q12	364	192	192	192
   q13	18099	3477	3423	3423
   q14	257	220	218	218
   q15	562	515	511	511
   q16	461	395	377	377
   q17	944	510	500	500
   q18	7656	7012	6939	6939
   q19	1612	1304	1355	1304
   q20	538	307	293	293
   q21	2818	2534	2450	2450
   q22	389	328	316	316
   Total cold run time: 109383 ms
   Total hot run time: 40008 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	6009	5806	5675	5675
   q2	355	228	222	222
   q3	3284	3122	3115	3115
   q4	2251	2008	2089	2008
   q5	6068	5944	6022	5944
   q6	207	126	121	121
   q7	2455	2050	1973	1973
   q8	3705	3659	3874	3659
   q9	9728	25503	9606	9606
   q10	10511	3821	3901	3821
   q11	721	461	482	461
   q12	1498	658	673	658
   q13	17356	3256	3235	3235
   q14	394	256	274	256
   q15	568	509	514	509
   q16	564	477	473	473
   q17	9324	1962	1960	1960
   q18	9529	10413	9511	9511
   q19	14488	1573	1617	1573
   q20	4709	1956	1976	1956
   q21	16303	5616	5709	5616
   q22	3032	618	652	618
   Total cold run time: 123059 ms
   Total hot run time: 62970 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org