You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zddr (via GitHub)" <gi...@apache.org> on 2023/12/20 08:49:26 UTC
[PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
zddr opened a new pull request, #28723:
URL: https://github.com/apache/doris/pull/28723
## Proposed changes
Issue Number: close #xxx
fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864217339
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 3b61e300b5405ba6b7579e8dbce4865de19cd5ab, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4675 4414 4421 4414
q2 367 148 158 148
q3 1471 1272 1240 1240
q4 1102 933 870 870
q5 3154 3139 3170 3139
q6 251 128 128 128
q7 1009 490 485 485
q8 2201 2183 2182 2182
q9 6695 6637 6668 6637
q10 3220 3295 3290 3290
q11 315 191 191 191
q12 354 205 199 199
q13 4565 3794 3816 3794
q14 245 209 213 209
q15 570 526 528 526
q16 443 385 389 385
q17 1009 614 558 558
q18 7144 6956 6953 6953
q19 1532 1360 1397 1360
q20 503 315 309 309
q21 3074 2626 2690 2626
q22 346 276 283 276
Total cold run time: 44245 ms
Total hot run time: 39919 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4345 4338 4300 4300
q2 270 166 172 166
q3 3535 3531 3539 3531
q4 2391 2380 2365 2365
q5 5758 5728 5746 5728
q6 241 122 123 122
q7 2387 1876 1872 1872
q8 3507 3526 3521 3521
q9 9040 8989 8996 8989
q10 3903 4013 3996 3996
q11 495 374 374 374
q12 771 590 588 588
q13 4270 3601 3552 3552
q14 284 253 248 248
q15 568 522 520 520
q16 501 443 469 443
q17 1878 1836 1843 1836
q18 8648 8330 8329 8329
q19 1708 1739 1754 1739
q20 2240 1967 1934 1934
q21 6560 6202 6152 6152
q22 495 411 417 411
Total cold run time: 63795 ms
Total hot run time: 60716 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864328764
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864328718
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864127246
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.55 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17183412102 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28723:
URL: https://github.com/apache/doris/pull/28723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864213754
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.42 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17183680749 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864084220
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864157247
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28723:
URL: https://github.com/apache/doris/pull/28723#issuecomment-1864135039
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 87f5657f2f8d569b04c58ae5334bacccc61b9682, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4669 4392 4421 4392
q2 357 158 158 158
q3 1461 1231 1229 1229
q4 1113 932 906 906
q5 3156 3157 3166 3157
q6 252 132 129 129
q7 1001 485 492 485
q8 2197 2200 2176 2176
q9 6669 6675 6702 6675
q10 3204 3296 3285 3285
q11 310 187 189 187
q12 359 210 213 210
q13 4602 3860 3808 3808
q14 238 211 217 211
q15 563 514 531 514
q16 436 397 394 394
q17 992 585 589 585
q18 7189 6905 6779 6779
q19 1512 1374 1399 1374
q20 512 320 311 311
q21 3089 2675 2709 2675
q22 344 280 282 280
Total cold run time: 44225 ms
Total hot run time: 39920 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4354 4355 4363 4355
q2 267 168 175 168
q3 3552 3534 3522 3522
q4 2389 2370 2369 2369
q5 5739 5764 5739 5739
q6 239 123 121 121
q7 2367 1873 1894 1873
q8 3526 3547 3528 3528
q9 9050 9046 8993 8993
q10 3944 3989 4024 3989
q11 486 369 371 369
q12 776 589 614 589
q13 4297 3563 3575 3563
q14 291 261 252 252
q15 572 522 528 522
q16 521 446 469 446
q17 1878 1897 1833 1833
q18 8662 8074 8284 8074
q19 1767 1736 1747 1736
q20 2266 1944 1941 1941
q21 6542 6202 6157 6157
q22 515 415 417 415
Total cold run time: 64000 ms
Total hot run time: 60554 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org