You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2024/03/04 17:09:38 UTC

[PR] [fix](move-memtable) free resources when fail to open stream [doris]

kaijchen opened a new pull request, #31762:
URL: https://github.com/apache/doris/pull/31762

   ## Proposed changes
   
   Free resources when error occurs in load stream open.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1978366886

   TeamCity be ut coverage result:
    Function Coverage: 35.43% (8557/24152) 
    Line Coverage: 27.19% (69436/255406)
    Region Coverage: 26.41% (36032/136426)
    Branch Coverage: 23.25% (18412/79202)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/146dfa5fd082d15296da623432cf71954b6f7aa9_146dfa5fd082d15296da623432cf71954b6f7aa9/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 merged PR #31762:
URL: https://github.com/apache/doris/pull/31762


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977072041

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977152332

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 146dfa5fd082d15296da623432cf71954b6f7aa9 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       18.1 seconds inserted 10000000 Rows, about 552K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977876366

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977138540

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 177845 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 146dfa5fd082d15296da623432cf71954b6f7aa9, data reload: false
   
   query1	940	370	345	345
   query2	7351	2191	2049	2049
   query3	6698	221	220	220
   query4	25830	20998	20979	20979
   query5	4307	2703	476	476
   query6	279	184	174	174
   query7	4635	309	307	307
   query8	236	179	179	179
   query9	8487	2169	2172	2169
   query10	450	236	246	236
   query11	15028	14371	14445	14371
   query12	137	90	85	85
   query13	1632	424	432	424
   query14	9085	7013	7014	7013
   query15	246	198	192	192
   query16	7239	271	283	271
   query17	962	617	577	577
   query18	1927	294	286	286
   query19	203	160	160	160
   query20	97	85	93	85
   query21	202	132	127	127
   query22	4688	4452	4411	4411
   query23	31517	30972	30457	30457
   query24	12244	3027	3043	3027
   query25	704	399	398	398
   query26	1931	161	173	161
   query27	3003	375	375	375
   query28	6597	1827	1799	1799
   query29	1328	624	608	608
   query30	312	151	147	147
   query31	940	722	726	722
   query32	104	65	61	61
   query33	763	272	265	265
   query34	1008	467	502	467
   query35	927	824	792	792
   query36	940	826	814	814
   query37	288	70	70	70
   query38	3282	3155	3140	3140
   query39	1427	1355	1363	1355
   query40	298	130	142	130
   query41	61	57	57	57
   query42	112	112	109	109
   query43	452	403	395	395
   query44	1078	732	711	711
   query45	204	199	193	193
   query46	1070	798	760	760
   query47	1622	1556	1555	1555
   query48	413	351	344	344
   query49	1183	329	328	328
   query50	785	380	375	375
   query51	6668	6536	6573	6536
   query52	105	94	100	94
   query53	355	291	286	286
   query54	339	229	241	229
   query55	93	85	80	80
   query56	231	238	227	227
   query57	1101	990	1004	990
   query58	249	205	216	205
   query59	2587	2361	2370	2361
   query60	255	243	238	238
   query61	95	92	91	91
   query62	658	418	396	396
   query63	309	274	281	274
   query64	6253	3283	3134	3134
   query65	3018	3063	2999	2999
   query66	1475	332	323	323
   query67	14875	14903	14394	14394
   query68	7759	588	616	588
   query69	651	390	407	390
   query70	1239	1146	1119	1119
   query71	612	300	273	273
   query72	8660	2813	2608	2608
   query73	801	327	341	327
   query74	7198	6850	6987	6850
   query75	4648	2673	2691	2673
   query76	5836	1113	1194	1113
   query77	850	263	262	262
   query78	10324	9670	9498	9498
   query79	8479	527	527	527
   query80	1125	426	417	417
   query81	562	221	219	219
   query82	744	97	95	95
   query83	206	152	155	152
   query84	296	79	87	79
   query85	1645	350	331	331
   query86	514	277	304	277
   query87	3417	3148	3186	3148
   query88	4137	2401	2375	2375
   query89	576	380	378	378
   query90	2113	180	179	179
   query91	165	129	130	129
   query92	67	55	50	50
   query93	6941	534	508	508
   query94	1331	188	193	188
   query95	447	349	362	349
   query96	599	264	265	264
   query97	3993	3878	3869	3869
   query98	228	221	219	219
   query99	1203	785	783	783
   Total cold run time: 301284 ms
   Total hot run time: 177845 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977118490

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38127 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 146dfa5fd082d15296da623432cf71954b6f7aa9, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18534	4659	4181	4181
   q2	2263	172	142	142
   q3	11590	963	975	963
   q4	4970	968	961	961
   q5	8258	2804	2979	2804
   q6	192	122	125	122
   q7	1304	839	811	811
   q8	9496	2094	2078	2078
   q9	7380	6494	6480	6480
   q10	8239	2536	2584	2536
   q11	425	221	222	221
   q12	802	322	317	317
   q13	17972	2956	2969	2956
   q14	288	255	248	248
   q15	492	448	442	442
   q16	490	385	401	385
   q17	955	899	857	857
   q18	6784	5988	5866	5866
   q19	1591	1523	1532	1523
   q20	559	289	284	284
   q21	7582	3710	3667	3667
   q22	791	290	283	283
   Total cold run time: 110957 ms
   Total hot run time: 38127 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4010	3988	4035	3988
   q2	317	219	221	219
   q3	2962	2937	2888	2888
   q4	1856	1808	1822	1808
   q5	5208	5250	5202	5202
   q6	201	114	114	114
   q7	2310	1858	1845	1845
   q8	3190	3252	3277	3252
   q9	8523	8488	8527	8488
   q10	6099	3728	3737	3728
   q11	526	436	447	436
   q12	691	545	529	529
   q13	13558	2806	2791	2791
   q14	272	257	267	257
   q15	484	437	434	434
   q16	455	420	416	416
   q17	1688	1675	1668	1668
   q18	7756	7502	7275	7275
   q19	1611	1608	1615	1608
   q20	1925	1722	1764	1722
   q21	5004	4816	4795	4795
   q22	483	484	464	464
   Total cold run time: 69129 ms
   Total hot run time: 53927 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977088587

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977876285

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977147346

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.93 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 146dfa5fd082d15296da623432cf71954b6f7aa9, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.07	0.03	0.03
   query3	0.23	0.06	0.06
   query4	1.64	0.10	0.09
   query5	0.50	0.49	0.50
   query6	1.23	0.66	0.69
   query7	0.02	0.02	0.01
   query8	0.05	0.03	0.03
   query9	0.56	0.52	0.51
   query10	0.57	0.57	0.57
   query11	0.13	0.10	0.10
   query12	0.12	0.10	0.10
   query13	0.58	0.57	0.56
   query14	0.77	0.74	0.75
   query15	0.83	0.81	0.81
   query16	0.37	0.37	0.37
   query17	0.95	1.01	0.96
   query18	0.25	0.25	0.25
   query19	1.82	1.71	1.71
   query20	0.02	0.01	0.02
   query21	15.43	0.61	0.59
   query22	3.37	4.49	2.56
   query23	17.43	1.13	0.97
   query24	2.30	0.27	0.21
   query25	0.16	0.07	0.04
   query26	0.18	0.13	0.14
   query27	0.03	0.03	0.03
   query28	12.47	0.84	0.87
   query29	12.60	3.51	3.34
   query30	0.66	0.57	0.55
   query31	2.80	0.33	0.34
   query32	3.36	0.45	0.43
   query33	2.97	2.91	2.91
   query34	15.51	4.32	4.31
   query35	4.33	4.30	4.30
   query36	1.09	1.03	1.01
   query37	0.07	0.05	0.05
   query38	0.05	0.03	0.03
   query39	0.02	0.02	0.02
   query40	0.18	0.13	0.14
   query41	0.08	0.03	0.02
   query42	0.03	0.02	0.01
   query43	0.03	0.03	0.03
   Total cold run time: 105.9 s
   Total hot run time: 30.93 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) free resources when fail to open stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31762:
URL: https://github.com/apache/doris/pull/31762#issuecomment-1977070698

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org