You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2023/12/18 16:51:05 UTC
[PR] [fix](load) fix orphan mem limiter too large [doris]
kaijchen opened a new pull request, #28600:
URL: https://github.com/apache/doris/pull/28600
## Proposed changes
After #25205, memtable insert mem tracker is smaller than allocated size (the actual consumption).
The orphan mem tracker limiter becomes too large, causing memtable memory limiter to be inaccurate.
This PR fix the problem by multiplying input block size with a factor `memtable_insert_memory_ratio`.
So the memtable insert mem tracker could be near the actual mem consumption.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861057712
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861036322
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861230466
TeamCity be ut coverage result:
Function Coverage: 36.49% (8537/23395)
Line Coverage: 28.65% (69493/242570)
Region Coverage: 27.67% (35897/129755)
Branch Coverage: 24.40% (18340/75166)
Coverage Report: http://coverage.selectdb-in.cc/coverage/297de6052427fbda9997d266812520856c02b87e_297de6052427fbda9997d266812520856c02b87e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861297797
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 297de6052427fbda9997d266812520856c02b87e, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4710 4474 4480 4474
q2 368 146 159 146
q3 1452 1270 1283 1270
q4 1120 915 878 878
q5 3124 3145 3124 3124
q6 244 129 130 129
q7 982 499 477 477
q8 2219 2227 2182 2182
q9 6652 6652 6662 6652
q10 3216 3276 3255 3255
q11 329 202 199 199
q12 350 200 204 200
q13 4573 3809 3785 3785
q14 240 212 222 212
q15 566 525 517 517
q16 443 387 378 378
q17 992 572 497 497
q18 7258 6918 7124 6918
q19 1549 1423 1393 1393
q20 564 334 323 323
q21 3080 2629 2659 2629
q22 352 279 284 279
Total cold run time: 44383 ms
Total hot run time: 39917 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4394 4388 4383 4383
q2 268 165 167 165
q3 3538 3531 3508 3508
q4 2378 2367 2355 2355
q5 5722 5747 5707 5707
q6 242 124 122 122
q7 2394 1856 1872 1856
q8 3520 3514 3528 3514
q9 9010 9014 8951 8951
q10 3909 3978 3995 3978
q11 509 378 375 375
q12 769 606 590 590
q13 4287 3582 3579 3579
q14 284 256 264 256
q15 567 517 530 517
q16 507 446 457 446
q17 1888 1846 1863 1846
q18 8755 8290 8302 8290
q19 1720 1761 1730 1730
q20 2249 1965 1941 1941
q21 6506 6178 6107 6107
q22 502 449 445 445
Total cold run time: 63918 ms
Total hot run time: 60661 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861329395
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.61 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17188020447 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #28600:
URL: https://github.com/apache/doris/pull/28600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861954720
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](load) fix orphan mem limiter too large [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28600:
URL: https://github.com/apache/doris/pull/28600#issuecomment-1861954761
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org