You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "luwei16 (via GitHub)" <gi...@apache.org> on 2024/03/12 08:17:55 UTC

[PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

luwei16 opened a new pull request, #32113:
URL: https://github.com/apache/doris/pull/32113

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991352139

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "luwei16 (via GitHub)" <gi...@apache.org>.
luwei16 commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991012195

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991189399

   TeamCity be ut coverage result:
    Function Coverage: 34.97% (8555/24464) 
    Line Coverage: 26.81% (69524/259355)
    Region Coverage: 26.05% (36073/138472)
    Branch Coverage: 23.03% (18442/80094)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/41eaa0c67398ea3f69deb846528a26767b45ad54_41eaa0c67398ea3f69deb846528a26767b45ad54/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991011614

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991076137

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.54 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
   
   query1	0.04	0.04	0.04
   query2	0.05	0.02	0.02
   query3	0.22	0.06	0.06
   query4	1.67	0.10	0.10
   query5	0.51	0.51	0.52
   query6	1.19	0.63	0.63
   query7	0.02	0.02	0.01
   query8	0.04	0.02	0.02
   query9	0.56	0.48	0.49
   query10	0.53	0.54	0.55
   query11	0.12	0.09	0.09
   query12	0.11	0.10	0.09
   query13	0.57	0.57	0.57
   query14	0.74	0.75	0.75
   query15	0.78	0.78	0.77
   query16	0.39	0.40	0.40
   query17	0.98	1.00	0.99
   query18	0.20	0.28	0.25
   query19	1.77	1.75	1.71
   query20	0.01	0.01	0.01
   query21	15.42	0.56	0.54
   query22	2.44	1.51	2.22
   query23	17.13	0.87	0.72
   query24	3.17	1.29	0.51
   query25	0.20	0.08	0.03
   query26	0.66	0.12	0.13
   query27	0.03	0.03	0.04
   query28	11.39	0.85	0.85
   query29	12.59	3.25	3.22
   query30	0.60	0.56	0.63
   query31	2.80	0.34	0.33
   query32	3.39	0.44	0.44
   query33	2.82	2.79	2.80
   query34	16.15	4.31	4.32
   query35	4.40	4.28	4.27
   query36	1.07	1.01	1.00
   query37	0.07	0.05	0.05
   query38	0.03	0.03	0.03
   query39	0.03	0.02	0.01
   query40	0.17	0.13	0.13
   query41	0.07	0.02	0.01
   query42	0.03	0.02	0.02
   query43	0.03	0.03	0.02
   Total cold run time: 105.19 s
   Total hot run time: 29.54 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32113:
URL: https://github.com/apache/doris/pull/32113


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991082109

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991046677

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 36983 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17643	4464	4257	4257
   q2	3014	152	146	146
   q3	11305	918	923	918
   q4	5197	717	669	669
   q5	7818	2900	2912	2900
   q6	191	131	129	129
   q7	1214	855	842	842
   q8	9543	2109	2028	2028
   q9	7368	6458	6446	6446
   q10	8200	2467	2488	2467
   q11	420	224	214	214
   q12	788	293	286	286
   q13	17988	3001	3011	3001
   q14	279	247	259	247
   q15	496	455	454	454
   q16	480	394	413	394
   q17	954	583	558	558
   q18	6586	5873	5777	5777
   q19	1578	1428	1473	1428
   q20	551	306	298	298
   q21	7384	3273	3232	3232
   q22	787	292	303	292
   Total cold run time: 109784 ms
   Total hot run time: 36983 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4106	4067	4073	4067
   q2	323	226	226	226
   q3	2975	2837	2833	2833
   q4	1817	1523	1531	1523
   q5	5181	5231	5241	5231
   q6	198	120	120	120
   q7	2239	1863	1852	1852
   q8	3152	3273	3254	3254
   q9	8516	8542	8592	8542
   q10	6177	3685	3631	3631
   q11	537	440	432	432
   q12	721	516	554	516
   q13	8082	2854	2841	2841
   q14	271	243	251	243
   q15	485	448	440	440
   q16	454	398	404	398
   q17	1742	1465	1451	1451
   q18	7674	7243	7137	7137
   q19	6107	1501	1536	1501
   q20	1908	1715	1701	1701
   q21	4824	4777	4556	4556
   q22	499	457	448	448
   Total cold run time: 67988 ms
   Total hot run time: 52943 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991066837

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 186625 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 41eaa0c67398ea3f69deb846528a26767b45ad54, data reload: false
   
   query1	934	372	343	343
   query2	7405	2080	2023	2023
   query3	6714	215	210	210
   query4	31083	20779	20552	20552
   query5	4324	399	411	399
   query6	266	172	167	167
   query7	4634	305	295	295
   query8	227	165	166	165
   query9	9949	2306	2282	2282
   query10	442	225	231	225
   query11	16933	14102	14173	14102
   query12	130	81	79	79
   query13	1618	424	429	424
   query14	11063	10849	10666	10666
   query15	271	211	194	194
   query16	8062	273	267	267
   query17	1923	574	539	539
   query18	2029	284	286	284
   query19	243	158	156	156
   query20	93	85	86	85
   query21	203	124	127	124
   query22	4613	4398	4383	4383
   query23	31316	30844	30863	30844
   query24	11440	2762	2838	2762
   query25	632	378	384	378
   query26	1682	155	159	155
   query27	3080	345	348	345
   query28	8088	1926	1908	1908
   query29	998	660	625	625
   query30	308	142	145	142
   query31	978	719	722	719
   query32	88	64	53	53
   query33	760	252	249	249
   query34	1071	464	464	464
   query35	858	602	600	600
   query36	993	872	892	872
   query37	120	80	89	80
   query38	3280	3145	3141	3141
   query39	1427	1517	1404	1404
   query40	289	110	111	110
   query41	50	48	46	46
   query42	103	93	104	93
   query43	498	473	461	461
   query44	1344	706	689	689
   query45	280	266	254	254
   query46	1088	679	685	679
   query47	1643	1591	1613	1591
   query48	445	394	357	357
   query49	1198	332	326	326
   query50	761	376	372	372
   query51	6652	6546	6515	6515
   query52	105	89	97	89
   query53	341	284	318	284
   query54	318	245	248	245
   query55	84	85	84	84
   query56	246	222	232	222
   query57	1103	1038	1001	1001
   query58	235	203	208	203
   query59	2828	2641	2556	2556
   query60	272	262	262	262
   query61	113	112	114	112
   query62	607	402	396	396
   query63	296	282	271	271
   query64	6638	3996	3986	3986
   query65	3107	3027	3020	3020
   query66	1392	349	317	317
   query67	15151	14373	14222	14222
   query68	8801	541	552	541
   query69	620	375	382	375
   query70	1275	1115	1059	1059
   query71	514	269	276	269
   query72	6696	2681	2483	2483
   query73	745	320	324	320
   query74	8509	6625	6681	6625
   query75	9518	8054	7936	7936
   query76	5284	962	900	900
   query77	640	268	261	261
   query78	10345	9579	9520	9520
   query79	11809	504	511	504
   query80	1810	366	370	366
   query81	500	205	216	205
   query82	659	196	205	196
   query83	233	141	147	141
   query84	284	84	72	72
   query85	1278	319	307	307
   query86	381	308	315	308
   query87	3412	3284	3234	3234
   query88	5031	2356	2363	2356
   query89	510	356	358	356
   query90	2061	178	175	175
   query91	160	129	128	128
   query92	52	46	48	46
   query93	7348	510	498	498
   query94	1276	174	171	171
   query95	421	326	316	316
   query96	594	276	265	265
   query97	4421	4258	4253	4253
   query98	237	216	211	211
   query99	1146	762	760	760
   Total cold run time: 324035 ms
   Total hot run time: 186625 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](merge-cloud) calculate delete bitmap during cloud full compaction [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32113:
URL: https://github.com/apache/doris/pull/32113#issuecomment-1991296848

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org