You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "w41ter (via GitHub)" <gi...@apache.org> on 2024/02/05 03:15:08 UTC

[PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

w41ter opened a new pull request, #30833:
URL: https://github.com/apache/doris/pull/30833

   ## Proposed changes
   
   Issue Number: close #xxx
   
   In cloud mode, both features are not supported.
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926173894

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37526 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18358	4728	4589	4589
   q2	2623	145	149	145
   q3	11140	948	943	943
   q4	5021	752	744	744
   q5	7805	2916	2940	2916
   q6	187	123	125	123
   q7	1151	773	744	744
   q8	9703	2034	2030	2030
   q9	7283	6368	6397	6368
   q10	8107	2410	2443	2410
   q11	426	217	206	206
   q12	722	286	291	286
   q13	17994	3281	3334	3281
   q14	307	252	253	252
   q15	539	492	488	488
   q16	476	412	408	408
   q17	945	532	499	499
   q18	6745	6069	5984	5984
   q19	1559	1380	1387	1380
   q20	619	341	345	341
   q21	7025	3201	3086	3086
   q22	811	325	303	303
   Total cold run time: 109546 ms
   Total hot run time: 37526 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4516	4398	4472	4398
   q2	332	238	234	234
   q3	3014	2906	2778	2778
   q4	1857	1622	1604	1604
   q5	5205	5228	5271	5228
   q6	192	114	115	114
   q7	2140	1743	1819	1743
   q8	3148	3264	3250	3250
   q9	8364	8362	8345	8345
   q10	5972	3540	3555	3540
   q11	553	469	460	460
   q12	763	576	573	573
   q13	7034	3148	3132	3132
   q14	285	244	263	244
   q15	537	492	493	492
   q16	536	484	489	484
   q17	1842	1701	1742	1701
   q18	7938	7733	7608	7608
   q19	7152	1508	1494	1494
   q20	2152	1925	1883	1883
   q21	4803	4527	4573	4527
   q22	577	470	472	470
   Total cold run time: 68912 ms
   Total hot run time: 54302 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926181786

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 174961 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
   
   query1	923	348	342	342
   query2	6524	2095	2122	2095
   query3	6708	212	201	201
   query4	31292	22129	22018	22018
   query5	4308	429	357	357
   query6	259	162	166	162
   query7	4607	303	285	285
   query8	238	175	178	175
   query9	9120	2318	2318	2318
   query10	416	218	204	204
   query11	18853	15450	15318	15318
   query12	130	73	72	72
   query13	1612	423	436	423
   query14	8729	6870	6783	6783
   query15	234	171	179	171
   query16	8101	252	253	252
   query17	1859	533	497	497
   query18	2103	287	262	262
   query19	189	140	136	136
   query20	81	73	76	73
   query21	196	125	127	125
   query22	4844	4693	4662	4662
   query23	30923	30115	30125	30115
   query24	11698	2776	2772	2772
   query25	609	341	341	341
   query26	1803	153	147	147
   query27	3074	305	311	305
   query28	7768	1870	1849	1849
   query29	986	595	603	595
   query30	289	133	140	133
   query31	897	726	703	703
   query32	88	52	53	52
   query33	735	217	233	217
   query34	1116	465	468	465
   query35	860	776	763	763
   query36	1025	926	951	926
   query37	171	62	64	62
   query38	3260	3096	3133	3096
   query39	1306	1258	1259	1258
   query40	271	92	98	92
   query41	37	35	34	34
   query42	106	91	93	91
   query43	550	506	509	506
   query44	1084	690	705	690
   query45	191	182	171	171
   query46	1066	656	646	646
   query47	1583	1536	1535	1535
   query48	411	355	354	354
   query49	1216	289	288	288
   query50	756	384	376	376
   query51	5291	5200	5174	5174
   query52	103	88	83	83
   query53	336	272	271	271
   query54	289	222	215	215
   query55	78	75	74	74
   query56	229	211	202	202
   query57	1010	897	899	897
   query58	215	182	173	173
   query59	2442	2348	2468	2348
   query60	246	216	210	210
   query61	84	89	82	82
   query62	665	349	341	341
   query63	295	278	285	278
   query64	6392	3718	3499	3499
   query65	3262	3231	3206	3206
   query66	1353	306	311	306
   query67	14267	14165	14058	14058
   query68	4290	517	540	517
   query69	446	339	334	334
   query70	1292	1261	1225	1225
   query71	324	256	244	244
   query72	5981	2838	2680	2680
   query73	709	329	326	326
   query74	6706	6239	6317	6239
   query75	3014	2369	2333	2333
   query76	2513	871	963	871
   query77	316	226	233	226
   query78	9100	8834	8650	8650
   query79	2461	488	506	488
   query80	1999	347	342	342
   query81	520	196	191	191
   query82	760	82	85	82
   query83	246	130	124	124
   query84	283	78	91	78
   query85	2206	332	325	325
   query86	484	294	307	294
   query87	3377	3253	3222	3222
   query88	3882	2318	2329	2318
   query89	436	371	360	360
   query90	2007	164	163	163
   query91	155	120	121	120
   query92	59	46	41	41
   query93	2569	482	494	482
   query94	1293	175	175	175
   query95	7978	357	7776	357
   query96	603	266	284	266
   query97	4239	4120	4089	4089
   query98	210	205	193	193
   query99	1148	711	676	676
   Total cold run time: 293368 ms
   Total hot run time: 174961 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926162672

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926160209

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926174841

   TeamCity be ut coverage result:
    Function Coverage: 36.15% (8626/23861) 
    Line Coverage: 28.21% (70546/250066)
    Region Coverage: 27.21% (36405/133779)
    Branch Coverage: 24.00% (18642/77690)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/c88fe0bf4be64762c91582d27b3df4ea436ec920_c88fe0bf4be64762c91582d27b3df4ea436ec920/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926185164

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.09 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920, data reload: false
   
   query1	0.04	0.03	0.04
   query2	0.06	0.02	0.02
   query3	0.22	0.06	0.05
   query4	1.68	0.10	0.09
   query5	0.53	0.52	0.51
   query6	1.18	0.65	0.66
   query7	0.02	0.01	0.02
   query8	0.04	0.03	0.02
   query9	0.54	0.49	0.51
   query10	0.54	0.55	0.55
   query11	0.11	0.08	0.08
   query12	0.11	0.08	0.09
   query13	0.60	0.60	0.60
   query14	0.77	0.79	0.80
   query15	0.78	0.77	0.78
   query16	0.39	0.38	0.38
   query17	1.03	1.03	1.07
   query18	0.22	0.28	0.24
   query19	1.86	1.78	1.79
   query20	0.02	0.01	0.01
   query21	15.44	0.57	0.56
   query22	2.59	1.71	1.78
   query23	17.40	0.96	0.69
   query24	2.71	1.66	0.26
   query25	0.32	0.25	0.12
   query26	0.58	0.14	0.15
   query27	0.05	0.05	0.04
   query28	11.60	0.83	0.85
   query29	12.57	3.22	3.21
   query30	0.64	0.53	0.54
   query31	2.78	0.33	0.36
   query32	3.38	0.48	0.49
   query33	3.22	3.20	3.23
   query34	15.76	4.26	4.24
   query35	4.32	4.27	4.30
   query36	1.10	1.05	1.07
   query37	0.07	0.05	0.05
   query38	0.04	0.03	0.03
   query39	0.02	0.01	0.01
   query40	0.16	0.13	0.12
   query41	0.07	0.02	0.01
   query42	0.03	0.01	0.02
   query43	0.03	0.02	0.02
   Total cold run time: 105.62 s
   Total hot run time: 30.09 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30833:
URL: https://github.com/apache/doris/pull/30833


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926159865

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926166228

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926162695

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](merge-cloud) Disable memtable on sink node & unique key merge on write [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30833:
URL: https://github.com/apache/doris/pull/30833#issuecomment-1926187213

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit c88fe0bf4be64762c91582d27b3df4ea436ec920 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org