You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/01/11 07:03:58 UTC

[PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

hust-hhb opened a new pull request, #29848:
URL: https://github.com/apache/doris/pull/29848

   wal file on materialized-view will have repeat column id, when relaying it, need to consider deduplication.
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886425496

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886505098

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.27 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
   
   query1	0.05	0.06	0.05
   query2	0.06	0.03	0.03
   query3	0.25	0.11	0.10
   query4	1.77	0.13	0.12
   query5	0.52	0.52	0.52
   query6	1.37	0.64	0.66
   query7	0.02	0.01	0.01
   query8	0.05	0.02	0.02
   query9	0.54	0.50	0.50
   query10	0.56	0.54	0.53
   query11	0.13	0.09	0.09
   query12	0.12	0.10	0.09
   query13	0.60	0.60	0.60
   query14	0.80	0.78	0.80
   query15	0.81	0.81	0.82
   query16	0.35	0.36	0.35
   query17	0.98	0.96	0.98
   query18	0.26	0.24	0.24
   query19	1.86	1.76	1.77
   query20	0.02	0.01	0.01
   query21	15.40	0.57	0.57
   query22	2.49	2.02	1.53
   query23	17.17	1.07	0.83
   query24	16.18	0.61	0.58
   query25	2.16	0.15	0.15
   query26	0.14	0.14	0.14
   query27	0.14	0.15	0.15
   query28	5.85	0.81	0.84
   query29	12.55	3.15	3.27
   query30	0.52	0.48	0.48
   query31	2.78	0.36	0.37
   query32	3.35	0.49	0.48
   query33	3.19	3.22	3.23
   query34	15.83	4.16	4.14
   query35	4.18	4.18	4.16
   query36	1.10	1.05	1.06
   query37	0.07	0.05	0.05
   query38	0.04	0.03	0.03
   query39	0.02	0.02	0.02
   query40	0.16	0.13	0.13
   query41	0.07	0.01	0.02
   query42	0.03	0.02	0.01
   query43	0.02	0.02	0.01
   Total cold run time: 114.56 s
   Total hot run time: 30.27 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886419908

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886534165

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 46.47 seconds
    stream load tsv:          560 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          21.4 seconds inserted 10000000 Rows, about 467K ops/s
    storage size: 17183782566 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886428110

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886440530

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38989 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17896	5044	4992	4992
   q2	2041	148	136	136
   q3	10830	1166	1206	1166
   q4	10614	839	779	779
   q5	8617	3246	3247	3246
   q6	209	128	125	125
   q7	905	524	516	516
   q8	9586	2026	2024	2024
   q9	7946	6636	6606	6606
   q10	8367	3066	3052	3052
   q11	418	202	201	201
   q12	359	189	195	189
   q13	18129	3448	3449	3448
   q14	245	214	212	212
   q15	556	520	501	501
   q16	429	373	377	373
   q17	948	497	552	497
   q18	7373	6799	6701	6701
   q19	1596	1366	1291	1291
   q20	532	313	294	294
   q21	2807	2355	2417	2355
   q22	347	298	285	285
   Total cold run time: 110750 ms
   Total hot run time: 38989 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4963	4951	4935	4935
   q2	313	205	203	203
   q3	3331	3303	3301	3301
   q4	2239	2245	2245	2245
   q5	5822	5790	5788	5788
   q6	191	118	116	116
   q7	2318	1860	1883	1860
   q8	3448	3576	3590	3576
   q9	8861	8875	8821	8821
   q10	3782	3771	3834	3771
   q11	563	441	418	418
   q12	788	608	605	605
   q13	3882	3234	3205	3205
   q14	289	267	271	267
   q15	563	519	513	513
   q16	503	473	465	465
   q17	2061	2035	2046	2035
   q18	8713	8333	8269	8269
   q19	1594	1624	1611	1611
   q20	2173	1932	1960	1932
   q21	6116	5643	5666	5643
   q22	540	456	473	456
   Total cold run time: 63053 ms
   Total hot run time: 60035 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886799451

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886481894

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 178327 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
   
   query1	954	339	336	336
   query2	6683	1965	2144	1965
   query3	6715	219	212	212
   query4	26099	22319	22032	22032
   query5	5874	517	493	493
   query6	267	189	185	185
   query7	4613	278	281	278
   query8	233	210	194	194
   query9	9128	2774	2777	2774
   query10	582	252	270	252
   query11	16118	15347	15320	15320
   query12	136	89	79	79
   query13	1615	347	347	347
   query14	11997	7393	7083	7083
   query15	251	192	188	188
   query16	6433	284	270	270
   query17	1915	494	497	494
   query18	1956	280	270	270
   query19	294	144	137	137
   query20	87	78	80	78
   query21	186	91	95	91
   query22	4971	4583	4803	4583
   query23	31659	30899	30946	30899
   query24	11990	2840	2807	2807
   query25	567	349	343	343
   query26	1703	156	151	151
   query27	2841	274	285	274
   query28	7148	1875	1869	1869
   query29	1715	398	385	385
   query30	295	149	155	149
   query31	962	800	781	781
   query32	99	65	58	58
   query33	749	263	267	263
   query34	901	450	444	444
   query35	864	734	740	734
   query36	1339	1272	1298	1272
   query37	104	70	71	70
   query38	3407	3319	3288	3288
   query39	1320	1289	1286	1286
   query40	314	95	93	93
   query41	40	37	36	36
   query42	103	92	97	92
   query43	556	534	526	526
   query44	1140	723	732	723
   query45	217	186	185	185
   query46	1063	657	689	657
   query47	1663	1583	1565	1565
   query48	386	307	321	307
   query49	1240	330	320	320
   query50	740	330	328	328
   query51	5252	5263	5218	5218
   query52	95	92	96	92
   query53	223	156	154	154
   query54	1417	560	599	560
   query55	95	81	91	81
   query56	217	191	205	191
   query57	1017	941	948	941
   query58	238	217	211	211
   query59	2765	2614	2621	2614
   query60	267	233	264	233
   query61	84	84	83	83
   query62	620	416	406	406
   query63	174	155	155	155
   query64	5815	1734	1822	1734
   query65	3335	3276	3262	3262
   query66	1142	341	327	327
   query67	15968	15346	15399	15346
   query68	12131	537	551	537
   query69	580	282	281	281
   query70	1729	1574	1645	1574
   query71	505	238	222	222
   query72	4966	2851	2839	2839
   query73	2422	332	330	330
   query74	6987	6411	6456	6411
   query75	5050	2266	2302	2266
   query76	6312	1110	1114	1110
   query77	877	289	270	270
   query78	9853	8719	8569	8569
   query79	1044	526	520	520
   query80	661	365	350	350
   query81	462	215	213	213
   query82	211	91	87	87
   query83	162	136	139	136
   query84	268	65	57	57
   query85	981	298	288	288
   query86	427	386	421	386
   query87	3574	3388	3384	3384
   query88	3175	2238	2270	2238
   query89	341	263	248	248
   query90	1878	216	208	208
   query91	170	135	148	135
   query92	64	53	54	53
   query93	1579	497	441	441
   query94	805	192	187	187
   query95	497	441	428	428
   query96	638	323	319	319
   query97	4320	4148	4170	4148
   query98	210	197	187	187
   query99	988	731	747	731
   Total cold run time: 294823 ms
   Total hot run time: 178327 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29848:
URL: https://github.com/apache/doris/pull/29848


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886514578

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       12.4 seconds inserted 10000000 Rows, about 806K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886446968

   TeamCity be ut coverage result:
    Function Coverage: 36.72% (8650/23558) 
    Line Coverage: 28.75% (70469/245123)
    Region Coverage: 27.67% (36406/131584)
    Branch Coverage: 24.38% (18629/76422)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/77368bc58ae230469b0407e97f07246ad1cdbd56_77368bc58ae230469b0407e97f07246ad1cdbd56/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org