You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2024/10/22 07:55:48 UTC

[PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

kaijchen opened a new pull request, #42249:
URL: https://github.com/apache/doris/pull/42249

   ## Proposed changes
   Currently, an upstream BE (sink_v2) will open multiple streams to a downstream BE (load_stream).
   If any of the streams fails, the use_cnt on the downstream BE will be messed up.
   The load_stream will not report any success tablets to the sink_v2 since in its view there are still unfinished streams.
   
   So fault tolerance when open streams is not meaningful in practical, and may cause data lost.
   i.e. Upstream think there is still working streams to transfer data, but downstream does not report any commit info.
   
   This PR removes fault tolerance when open multiple streams to the same backend.
   If any of the open fails, the upstream sink_v2 should mark the downstream BE as failed replicas.
   
   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42249:
URL: https://github.com/apache/doris/pull/42249#issuecomment-2428530447

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #42249:
URL: https://github.com/apache/doris/pull/42249#issuecomment-2428530977

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42249:
URL: https://github.com/apache/doris/pull/42249#issuecomment-2428604143

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 188471 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit e2814d6cadaa498d3a181b2d3d67063f5bda9937, data reload: false
   
   query1	997	364	361	361
   query2	6521	2099	2009	2009
   query3	6706	212	227	212
   query4	33996	23300	23642	23300
   query5	4367	452	441	441
   query6	263	171	166	166
   query7	4637	312	301	301
   query8	255	213	199	199
   query9	9587	2674	2664	2664
   query10	478	273	264	264
   query11	17969	15048	15156	15048
   query12	156	100	104	100
   query13	1624	417	394	394
   query14	9755	7103	6328	6328
   query15	250	170	171	170
   query16	8071	481	503	481
   query17	1628	564	554	554
   query18	2113	302	312	302
   query19	256	146	149	146
   query20	117	107	107	107
   query21	214	105	104	104
   query22	4445	4176	4207	4176
   query23	34725	33747	34027	33747
   query24	11707	2848	2756	2756
   query25	696	410	403	403
   query26	1772	167	170	167
   query27	2838	301	293	293
   query28	8022	2486	2439	2439
   query29	1037	438	450	438
   query30	332	156	160	156
   query31	1010	789	818	789
   query32	99	58	59	58
   query33	760	298	292	292
   query34	960	489	500	489
   query35	913	750	716	716
   query36	1083	929	917	917
   query37	186	89	87	87
   query38	3953	3933	3811	3811
   query39	1465	1425	1400	1400
   query40	295	102	98	98
   query41	53	50	48	48
   query42	116	97	94	94
   query43	522	483	486	483
   query44	1261	764	791	764
   query45	194	166	165	165
   query46	1122	721	714	714
   query47	1882	1816	1786	1786
   query48	469	373	363	363
   query49	1257	396	385	385
   query50	815	415	397	397
   query51	7006	6865	6802	6802
   query52	105	96	93	93
   query53	259	188	179	179
   query54	1204	468	461	461
   query55	80	74	78	74
   query56	276	250	257	250
   query57	1203	1087	1086	1086
   query58	276	246	246	246
   query59	3075	2814	2818	2814
   query60	311	282	279	279
   query61	125	124	134	124
   query62	837	650	678	650
   query63	206	181	182	181
   query64	5202	638	617	617
   query65	3274	3308	3158	3158
   query66	1422	313	293	293
   query67	15714	15279	15115	15115
   query68	4860	557	548	548
   query69	535	285	286	285
   query70	1137	1066	1124	1066
   query71	452	279	281	279
   query72	7372	3943	3816	3816
   query73	760	343	362	343
   query74	10435	8891	8926	8891
   query75	3967	2645	2638	2638
   query76	3427	870	930	870
   query77	554	276	291	276
   query78	9836	9396	9423	9396
   query79	3083	578	590	578
   query80	2338	441	446	441
   query81	558	235	243	235
   query82	937	143	146	143
   query83	289	134	133	133
   query84	300	87	89	87
   query85	2195	291	284	284
   query86	483	310	303	303
   query87	4457	4325	4430	4325
   query88	3987	2413	2399	2399
   query89	402	280	285	280
   query90	1959	184	183	183
   query91	173	140	141	140
   query92	67	48	49	48
   query93	2431	546	529	529
   query94	1036	297	287	287
   query95	349	251	244	244
   query96	612	280	281	280
   query97	3299	3167	3184	3167
   query98	213	194	197	194
   query99	1640	1300	1279	1279
   Total cold run time: 308667 ms
   Total hot run time: 188471 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 merged PR #42249:
URL: https://github.com/apache/doris/pull/42249


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](move-memtable) abstract multi-streams to one logical stream (#42039) [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42249:
URL: https://github.com/apache/doris/pull/42249#issuecomment-2428573818

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 40355 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e2814d6cadaa498d3a181b2d3d67063f5bda9937, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17658	7473	7408	7408
   q2	2128	153	143	143
   q3	10914	1101	1155	1101
   q4	10229	724	736	724
   q5	7721	2829	2758	2758
   q6	234	153	146	146
   q7	969	601	617	601
   q8	9343	1880	1951	1880
   q9	6515	6355	6347	6347
   q10	6996	2292	2242	2242
   q11	436	241	254	241
   q12	404	213	213	213
   q13	17811	3001	2964	2964
   q14	235	219	209	209
   q15	568	531	534	531
   q16	672	601	602	601
   q17	955	536	521	521
   q18	7318	6611	6517	6517
   q19	1834	960	991	960
   q20	571	283	271	271
   q21	3834	3178	3008	3008
   q22	1026	991	969	969
   Total cold run time: 108371 ms
   Total hot run time: 40355 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	7349	7253	7246	7246
   q2	320	224	232	224
   q3	2858	2677	2761	2677
   q4	1869	1707	1708	1707
   q5	5355	5444	5453	5444
   q6	231	143	141	141
   q7	2102	1645	1654	1645
   q8	3198	3380	3411	3380
   q9	8501	8472	8555	8472
   q10	3427	3370	3392	3370
   q11	580	474	479	474
   q12	760	555	587	555
   q13	16882	2968	2980	2968
   q14	285	264	256	256
   q15	599	564	518	518
   q16	689	657	666	657
   q17	1797	1564	1556	1556
   q18	7617	7464	7247	7247
   q19	1641	1479	1442	1442
   q20	1971	1794	1796	1794
   q21	5236	5029	5024	5024
   q22	1083	978	971	971
   Total cold run time: 74350 ms
   Total hot run time: 57768 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org