You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2025/10/14 07:44:12 UTC

[PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

sollhui opened a new pull request, #56948:
URL: https://github.com/apache/doris/pull/56948

   ### What problem does this PR solve?
   
   In some cases, the size of `segments` may be relatively small, which not only affects read performance but may also cause transaction commit failures in cloud mode for fdb value limit.
   
   We introduce the adaptive memtable write buffer size scheme, which adaptively increases the write buffer size when there is enough memory to avoid the generation of small segments as much as possible.
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405552691

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.9 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
   
   query1	0.05	0.05	0.05
   query2	0.10	0.05	0.06
   query3	0.25	0.08	0.08
   query4	1.61	0.12	0.12
   query5	0.27	0.28	0.24
   query6	1.19	0.64	0.64
   query7	0.02	0.03	0.03
   query8	0.05	0.04	0.04
   query9	0.62	0.53	0.53
   query10	0.58	0.60	0.58
   query11	0.17	0.11	0.11
   query12	0.16	0.12	0.12
   query13	0.64	0.63	0.62
   query14	1.02	1.04	1.04
   query15	0.90	0.86	0.84
   query16	0.40	0.40	0.43
   query17	1.05	1.08	1.05
   query18	0.22	0.20	0.20
   query19	1.93	1.80	1.85
   query20	0.02	0.01	0.02
   query21	15.47	0.92	0.59
   query22	0.76	1.22	0.62
   query23	14.92	1.39	0.62
   query24	7.00	1.20	0.36
   query25	0.32	0.19	0.11
   query26	0.58	0.16	0.13
   query27	0.08	0.06	0.06
   query28	9.32	1.37	0.92
   query29	12.58	3.89	3.24
   query30	0.29	0.15	0.11
   query31	2.84	0.59	0.39
   query32	3.26	0.58	0.48
   query33	3.12	3.10	3.09
   query34	16.26	5.50	4.86
   query35	5.00	4.87	4.93
   query36	0.71	0.52	0.52
   query37	0.10	0.08	0.07
   query38	0.06	0.05	0.04
   query39	0.03	0.03	0.03
   query40	0.19	0.16	0.14
   query41	0.09	0.04	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.04	0.04
   Total cold run time: 104.31 s
   Total hot run time: 29.9 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401391595

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.63 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
   
   query1	0.05	0.05	0.04
   query2	0.10	0.06	0.05
   query3	0.25	0.09	0.08
   query4	1.60	0.12	0.11
   query5	0.27	0.26	0.25
   query6	1.18	0.66	0.63
   query7	0.03	0.03	0.02
   query8	0.06	0.04	0.05
   query9	0.62	0.52	0.52
   query10	0.57	0.61	0.57
   query11	0.16	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.64	0.61	0.61
   query14	1.00	1.02	1.02
   query15	0.86	0.85	0.86
   query16	0.39	0.40	0.39
   query17	1.04	1.05	1.06
   query18	0.22	0.20	0.20
   query19	1.98	1.90	1.82
   query20	0.01	0.01	0.01
   query21	15.46	0.93	0.58
   query22	0.76	1.07	0.63
   query23	15.08	1.38	0.69
   query24	6.74	1.52	1.08
   query25	0.48	0.11	0.15
   query26	0.63	0.17	0.13
   query27	0.06	0.06	0.05
   query28	9.90	1.34	0.95
   query29	12.60	4.01	3.30
   query30	0.28	0.15	0.11
   query31	2.83	0.60	0.38
   query32	3.25	0.57	0.47
   query33	3.10	3.07	3.14
   query34	16.15	5.47	4.83
   query35	4.88	4.93	4.88
   query36	0.69	0.52	0.50
   query37	0.10	0.08	0.07
   query38	0.08	0.05	0.04
   query39	0.03	0.03	0.03
   query40	0.18	0.16	0.15
   query41	0.08	0.04	0.03
   query42	0.04	0.03	0.03
   query43	0.05	0.03	0.05
   Total cold run time: 104.63 s
   Total hot run time: 30.63 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405531581

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 188340 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
   
   query1	1075	422	411	411
   query2	6135	1724	1739	1724
   query3	4262	222	226	222
   query4	25767	23463	23029	23029
   query5	4546	640	486	486
   query6	314	237	228	228
   query7	4326	502	306	306
   query8	307	259	273	259
   query9	6778	2595	2561	2561
   query10	508	334	302	302
   query11	15537	15282	14762	14762
   query12	195	115	112	112
   query13	1649	549	428	428
   query14	11233	9318	9333	9318
   query15	216	204	201	201
   query16	7692	701	531	531
   query17	1606	798	676	676
   query18	2022	426	319	319
   query19	215	205	168	168
   query20	138	126	119	119
   query21	217	145	125	125
   query22	4113	4134	4127	4127
   query23	34047	33190	32989	32989
   query24	7988	2391	2385	2385
   query25	582	499	444	444
   query26	1239	271	160	160
   query27	2729	499	355	355
   query28	4416	2163	2179	2163
   query29	809	613	478	478
   query30	309	232	214	214
   query31	911	822	756	756
   query32	86	88	75	75
   query33	589	378	326	326
   query34	855	859	531	531
   query35	803	891	765	765
   query36	983	999	929	929
   query37	123	108	95	95
   query38	3566	3502	3454	3454
   query39	1478	1461	1482	1461
   query40	221	129	123	123
   query41	65	62	63	62
   query42	128	116	110	110
   query43	500	509	483	483
   query44	1386	850	814	814
   query45	195	185	179	179
   query46	883	1004	637	637
   query47	1786	1791	1709	1709
   query48	401	420	326	326
   query49	774	532	417	417
   query50	668	696	414	414
   query51	3918	3857	3860	3857
   query52	120	111	99	99
   query53	251	283	207	207
   query54	599	597	548	548
   query55	94	90	88	88
   query56	345	340	324	324
   query57	1182	1230	1110	1110
   query58	289	303	294	294
   query59	2646	2684	2519	2519
   query60	356	347	330	330
   query61	160	150	165	150
   query62	789	733	677	677
   query63	239	199	205	199
   query64	4519	1157	828	828
   query65	4054	3958	3944	3944
   query66	1081	517	357	357
   query67	15407	15189	14994	14994
   query68	8485	949	595	595
   query69	533	330	304	304
   query70	1412	1292	1282	1282
   query71	460	346	322	322
   query72	5755	4957	4884	4884
   query73	600	568	362	362
   query74	8862	8823	8880	8823
   query75	3355	3334	2812	2812
   query76	3571	1168	765	765
   query77	483	408	344	344
   query78	9743	9626	8835	8835
   query79	2608	838	595	595
   query80	631	565	500	500
   query81	502	279	256	256
   query82	254	236	149	149
   query83	265	269	244	244
   query84	257	113	96	96
   query85	885	469	429	429
   query86	379	304	323	304
   query87	3704	3756	3641	3641
   query88	3488	2296	2292	2292
   query89	399	332	308	308
   query90	1990	230	227	227
   query91	244	167	140	140
   query92	88	76	69	69
   query93	1970	1006	655	655
   query94	765	445	339	339
   query95	404	328	312	312
   query96	499	588	289	289
   query97	2960	2991	2869	2869
   query98	246	207	213	207
   query99	1339	1422	1298	1298
   Total cold run time: 270556 ms
   Total hot run time: 188340 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401653979

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400541307

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401779126

   # BE UT Coverage Report
   Increment line coverage `66.67% (10/15)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/41affff5c078c617e19c81ad1b586e085ba12fd6_41affff5c078c617e19c81ad1b586e085ba12fd6/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/41affff5c078c617e19c81ad1b586e085ba12fd6_41affff5c078c617e19c81ad1b586e085ba12fd6/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.55% (17755/33790) |
   | Line Coverage     | 37.75% (161468/427761) |
   | Region Coverage   | 32.20% (123179/382593) |
   | Branch Coverage   | 33.63% (54121/160935) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #56948:
URL: https://github.com/apache/doris/pull/56948


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401043994

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3406215926

   # BE Regression && UT Coverage Report
   Increment line coverage `70.97% (22/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/56948_697085113e3c0fa673b1f65c0c05c126551c7f66_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/56948_697085113e3c0fa673b1f65c0c05c126551c7f66_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.20% (23615/33169) |
   | Line Coverage     | 57.62% (246477/427792) |
   | Region Coverage   | 52.69% (204265/387666) |
   | Branch Coverage   | 54.53% (88281/161904) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401045254

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3408842754

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405104363

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401375037

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 190045 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
   
   query1	1095	436	416	416
   query2	6571	1704	1691	1691
   query3	6761	221	218	218
   query4	26886	23745	23090	23090
   query5	5828	644	498	498
   query6	342	255	221	221
   query7	4652	510	296	296
   query8	323	284	274	274
   query9	8708	2560	2547	2547
   query10	583	326	297	297
   query11	15342	15172	14973	14973
   query12	184	128	118	118
   query13	1718	560	458	458
   query14	12635	9195	9201	9195
   query15	254	192	175	175
   query16	7817	677	500	500
   query17	1571	788	705	705
   query18	2125	481	366	366
   query19	302	226	176	176
   query20	145	136	138	136
   query21	237	148	118	118
   query22	4804	4919	4576	4576
   query23	34661	33825	34070	33825
   query24	8359	2466	2527	2466
   query25	620	568	511	511
   query26	1734	285	163	163
   query27	2829	527	373	373
   query28	4392	2245	2216	2216
   query29	800	653	549	549
   query30	356	237	210	210
   query31	897	922	774	774
   query32	89	74	68	68
   query33	592	391	347	347
   query34	834	872	524	524
   query35	832	858	795	795
   query36	1013	1035	935	935
   query37	121	110	95	95
   query38	3729	3776	3718	3718
   query39	1565	1467	1496	1467
   query40	226	135	139	135
   query41	73	61	61	61
   query42	119	108	111	108
   query43	487	496	472	472
   query44	1314	830	829	829
   query45	193	187	176	176
   query46	844	1002	645	645
   query47	1812	1890	1731	1731
   query48	379	410	310	310
   query49	743	493	407	407
   query50	638	683	407	407
   query51	3915	3955	3869	3869
   query52	113	105	105	105
   query53	226	260	202	202
   query54	600	579	512	512
   query55	91	83	85	83
   query56	310	331	320	320
   query57	1176	1201	1142	1142
   query58	286	276	272	272
   query59	2511	2591	2515	2515
   query60	342	337	332	332
   query61	152	149	200	149
   query62	812	712	656	656
   query63	229	195	185	185
   query64	4409	1169	847	847
   query65	4036	3945	3951	3945
   query66	1037	415	329	329
   query67	15546	15509	15176	15176
   query68	6391	948	598	598
   query69	506	323	285	285
   query70	1371	1291	1311	1291
   query71	497	347	305	305
   query72	5850	4929	4869	4869
   query73	526	591	354	354
   query74	9425	8937	8498	8498
   query75	3757	3414	2813	2813
   query76	3209	1158	735	735
   query77	796	398	326	326
   query78	9518	9792	8942	8942
   query79	2747	799	606	606
   query80	668	571	469	469
   query81	523	271	229	229
   query82	224	161	139	139
   query83	280	260	242	242
   query84	256	109	102	102
   query85	875	481	405	405
   query86	341	305	284	284
   query87	3780	3763	3653	3653
   query88	4104	2209	2206	2206
   query89	377	324	293	293
   query90	2038	211	215	211
   query91	158	169	133	133
   query92	88	67	65	65
   query93	2323	971	643	643
   query94	639	438	349	349
   query95	396	321	309	309
   query96	489	573	279	279
   query97	2924	2984	2867	2867
   query98	251	214	214	214
   query99	1351	1401	1331	1331
   Total cold run time: 281603 ms
   Total hot run time: 190045 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3402140119

   # BE Regression && UT Coverage Report
   Increment line coverage `66.67% (10/15)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/56948_41affff5c078c617e19c81ad1b586e085ba12fd6_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/56948_41affff5c078c617e19c81ad1b586e085ba12fd6_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.14% (23565/33125) |
   | Line Coverage     | 57.56% (246066/427507) |
   | Region Coverage   | 52.52% (203549/387543) |
   | Branch Coverage   | 54.46% (88135/161824) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400517168

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400774541

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.06 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 18b7b6acd9549fdf87e638ad2d7f119c2245527b, data reload: false
   
   query1	0.05	0.05	0.05
   query2	0.10	0.05	0.06
   query3	0.25	0.09	0.09
   query4	1.63	0.13	0.13
   query5	0.29	0.28	0.26
   query6	1.27	0.68	0.64
   query7	0.03	0.03	0.03
   query8	0.06	0.05	0.05
   query9	0.64	0.54	0.52
   query10	0.60	0.58	0.58
   query11	0.19	0.11	0.12
   query12	0.15	0.12	0.12
   query13	0.63	0.62	0.62
   query14	1.01	1.01	1.04
   query15	0.87	0.89	0.87
   query16	0.40	0.40	0.39
   query17	1.03	1.09	1.04
   query18	0.21	0.20	0.20
   query19	1.91	1.89	1.87
   query20	0.02	0.01	0.01
   query21	15.42	0.89	0.57
   query22	0.77	1.20	0.61
   query23	15.01	1.43	0.68
   query24	6.85	1.39	0.38
   query25	0.53	0.26	0.12
   query26	0.62	0.16	0.14
   query27	0.07	0.05	0.06
   query28	8.87	1.39	0.92
   query29	12.60	3.89	3.22
   query30	0.28	0.14	0.11
   query31	2.82	0.63	0.38
   query32	3.24	0.55	0.48
   query33	3.14	3.08	3.13
   query34	16.03	5.42	4.84
   query35	4.92	4.95	4.93
   query36	0.68	0.52	0.51
   query37	0.11	0.07	0.08
   query38	0.07	0.05	0.05
   query39	0.04	0.03	0.03
   query40	0.18	0.16	0.15
   query41	0.09	0.04	0.03
   query42	0.04	0.03	0.03
   query43	0.04	0.04	0.04
   Total cold run time: 103.76 s
   Total hot run time: 30.06 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401654186

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #56948:
URL: https://github.com/apache/doris/pull/56948#discussion_r2428309679


##########
be/src/olap/memtable.h:
##########
@@ -213,6 +214,21 @@ class MemTable {
     // Used to wrapped by to_block to do exception handle logic
     Status _to_block(std::unique_ptr<vectorized::Block>* res);
 
+    int64_t _adaptive_write_buffer_size() const {
+        const int64_t current_load_mem_value = MemoryProfile::load_current_usage();
+        int64_t factor = 4;
+        // Memory usage intervals:
+        // (80 %, 100 %] → 1× buffer
+        // (50 %, 80 %]  → 2× buffer
+        // [0 %, 50 %]   → 4× buffer
+        if (current_load_mem_value > (_load_mem_limit * 4) / 5) { // > 80 %
+            factor = 1;
+        } else if (current_load_mem_value > _load_mem_limit / 2) { // > 50 %
+            factor = 2;
+        }
+        return config::write_buffer_size * factor;
+    }

Review Comment:
   put it to .cpp



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405980751

   # BE UT Coverage Report
   Increment line coverage `48.39% (15/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/697085113e3c0fa673b1f65c0c05c126551c7f66_697085113e3c0fa673b1f65c0c05c126551c7f66/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/697085113e3c0fa673b1f65c0c05c126551c7f66_697085113e3c0fa673b1f65c0c05c126551c7f66/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.57% (17789/33837) |
   | Line Coverage     | 37.77% (161665/428049) |
   | Region Coverage   | 32.20% (123249/382719) |
   | Branch Coverage   | 33.63% (54149/161015) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org