You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zddr (via GitHub)" <gi...@apache.org> on 2023/12/20 08:49:26 UTC

[PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

zddr opened a new pull request, #28723:
URL: https://github.com/apache/doris/pull/28723

   ## Proposed changes
   
   Issue Number: close #xxx
   
   fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864217339

   
   TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Tpch sf100 test result on commit 3b61e300b5405ba6b7579e8dbce4865de19cd5ab, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4675	4414	4421	4414
   q2	367	148	158	148
   q3	1471	1272	1240	1240
   q4	1102	933	870	870
   q5	3154	3139	3170	3139
   q6	251	128	128	128
   q7	1009	490	485	485
   q8	2201	2183	2182	2182
   q9	6695	6637	6668	6637
   q10	3220	3295	3290	3290
   q11	315	191	191	191
   q12	354	205	199	199
   q13	4565	3794	3816	3794
   q14	245	209	213	209
   q15	570	526	528	526
   q16	443	385	389	385
   q17	1009	614	558	558
   q18	7144	6956	6953	6953
   q19	1532	1360	1397	1360
   q20	503	315	309	309
   q21	3074	2626	2690	2626
   q22	346	276	283	276
   Total cold run time: 44245 ms
   Total hot run time: 39919 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4345	4338	4300	4300
   q2	270	166	172	166
   q3	3535	3531	3539	3531
   q4	2391	2380	2365	2365
   q5	5758	5728	5746	5728
   q6	241	122	123	122
   q7	2387	1876	1872	1872
   q8	3507	3526	3521	3521
   q9	9040	8989	8996	8989
   q10	3903	4013	3996	3996
   q11	495	374	374	374
   q12	771	590	588	588
   q13	4270	3601	3552	3552
   q14	284	253	248	248
   q15	568	522	520	520
   q16	501	443	469	443
   q17	1878	1836	1843	1836
   q18	8648	8330	8329	8329
   q19	1708	1739	1754	1739
   q20	2240	1967	1934	1934
   q21	6560	6202	6152	6152
   q22	495	411	417	411
   Total cold run time: 63795 ms
   Total hot run time: 60716 ms
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864328764

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864328718

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864127246

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.55 seconds
    stream load tsv:          563 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
    storage size: 17183412102 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28723:
URL: https://github.com/apache/doris/pull/28723


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864213754

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.42 seconds
    stream load tsv:          565 seconds loaded 74807831229 Bytes, about 126 MB/s
    stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
    stream load orc:          67 seconds loaded 1101869774 Bytes, about 15 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17183680749 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864084220

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864157247

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864135039

   
   TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Tpch sf100 test result on commit 87f5657f2f8d569b04c58ae5334bacccc61b9682, data reload: false
   
   run tpch-sf100 query with default conf and session variables
   q1	4669	4392	4421	4392
   q2	357	158	158	158
   q3	1461	1231	1229	1229
   q4	1113	932	906	906
   q5	3156	3157	3166	3157
   q6	252	132	129	129
   q7	1001	485	492	485
   q8	2197	2200	2176	2176
   q9	6669	6675	6702	6675
   q10	3204	3296	3285	3285
   q11	310	187	189	187
   q12	359	210	213	210
   q13	4602	3860	3808	3808
   q14	238	211	217	211
   q15	563	514	531	514
   q16	436	397	394	394
   q17	992	585	589	585
   q18	7189	6905	6779	6779
   q19	1512	1374	1399	1374
   q20	512	320	311	311
   q21	3089	2675	2709	2675
   q22	344	280	282	280
   Total cold run time: 44225 ms
   Total hot run time: 39920 ms
   
   run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
   q1	4354	4355	4363	4355
   q2	267	168	175	168
   q3	3552	3534	3522	3522
   q4	2389	2370	2369	2369
   q5	5739	5764	5739	5739
   q6	239	123	121	121
   q7	2367	1873	1894	1873
   q8	3526	3547	3528	3528
   q9	9050	9046	8993	8993
   q10	3944	3989	4024	3989
   q11	486	369	371	369
   q12	776	589	614	589
   q13	4297	3563	3575	3563
   q14	291	261	252	252
   q15	572	522	528	522
   q16	521	446	469	446
   q17	1878	1897	1833	1833
   q18	8662	8074	8284	8074
   q19	1767	1736	1747	1736
   q20	2266	1944	1941	1941
   q21	6542	6202	6157	6157
   q22	515	415	417	415
   Total cold run time: 64000 ms
   Total hot run time: 60554 ms
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org