You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2023/11/13 12:15:53 UTC
[PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
xinyiZzz opened a new pull request, #26904:
URL: https://github.com/apache/doris/pull/26904
## Proposed changes
1. Manually track query/load/compaction/etc. memory in Allocator instead of mem hook.
This will cause memory tracking loss for Query, but this is expected to be more controllable and does not require caching bthread local in pthread local for performance.
Use MemHook in the past:
Manual tracking in Allocator:
2. ThreadContext life cycle to manual control
In the past, ThreadContext was automatically created when it was used for the first time (this was usually in the Jemalloc Hook when the first malloc memory), and was automatically destroyed when the thread exited.
Now instead of manually controlling the create and destroy of ThreadContext, it is mainly created manually when the task thread start and destroyed before the task thread end.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808656062
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.56 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17160895403 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808842829
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 47284f0aaadada152de7523ed30f37d44b654a94, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5443 5187 5220 5187
q2 361 154 148 148
q3 2063 2018 2002 2002
q4 1414 1350 1367 1350
q5 3961 3976 3974 3974
q6 257 129 132 129
q7 1493 886 875 875
q8 2787 2774 2770 2770
q9 9651 9536 9451 9451
q10 3446 3592 3556 3556
q11 397 267 254 254
q12 438 278 279 278
q13 4665 4180 4142 4142
q14 318 294 283 283
q15 616 556 545 545
q16 676 598 592 592
q17 1113 1096 1090 1090
q18 8068 7473 7610 7473
q19 1666 1681 1680 1680
q20 580 328 298 298
q21 4699 4363 4376 4363
q22 522 408 431 408
Total cold run time: 54634 ms
Total hot run time: 50848 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5058 5066 5061 5061
q2 355 227 236 227
q3 4182 4026 4007 4007
q4 2845 2767 2782 2767
q5 9667 9635 9651 9635
q6 258 125 126 125
q7 3084 2440 2543 2440
q8 4824 4837 4799 4799
q9 13122 12976 12946 12946
q10 4076 4182 4183 4182
q11 755 662 645 645
q12 972 835 850 835
q13 4339 3852 3880 3852
q14 382 353 350 350
q15 622 562 550 550
q16 766 684 691 684
q17 3974 3889 3864 3864
q18 9620 9407 9416 9407
q19 1826 1773 1764 1764
q20 2473 2097 2055 2055
q21 8923 8651 8813 8651
q22 951 865 895 865
Total cold run time: 83074 ms
Total hot run time: 79711 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26904:
URL: https://github.com/apache/doris/pull/26904#discussion_r1391104993
##########
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##########
@@ -161,9 +163,9 @@ inline void ThreadMemTrackerMgr::pop_consumer_tracker() {
_consumer_tracker_stack.pop_back();
}
-inline void ThreadMemTrackerMgr::consume(int64_t size, bool large_memory_check) {
+inline void ThreadMemTrackerMgr::consume(int64_t size, int skip_large_memory_check) {
Review Comment:
why change the input parameter from bool to int?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808503682
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.05 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17161287639 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808262606
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808483795
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit ae12ab4456fad6363463fb501b2bf62e087ea33b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5306 5041 5156 5041
q2 353 147 157 147
q3 2074 2002 2035 2002
q4 1401 1366 1358 1358
q5 3957 3939 3990 3939
q6 252 129 127 127
q7 1434 865 883 865
q8 2812 2791 2770 2770
q9 9632 9543 9367 9367
q10 3454 3526 3527 3526
q11 389 267 255 255
q12 431 275 280 275
q13 4564 4130 4160 4130
q14 320 282 290 282
q15 615 572 547 547
q16 675 593 591 591
q17 1131 1090 1089 1089
q18 8073 7593 7480 7480
q19 1675 1698 1674 1674
q20 575 295 313 295
q21 4691 4363 4343 4343
q22 512 404 407 404
Total cold run time: 54326 ms
Total hot run time: 50507 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5069 4920 5023 4920
q2 330 241 228 228
q3 4073 4016 4027 4016
q4 2759 2764 2762 2762
q5 9529 9561 9578 9561
q6 252 120 122 120
q7 2682 2278 2231 2231
q8 4817 4820 4864 4820
q9 12991 12996 12956 12956
q10 4052 4182 4137 4137
q11 759 676 654 654
q12 1006 836 803 803
q13 4314 3864 3867 3864
q14 382 344 372 344
q15 612 536 524 524
q16 737 699 706 699
q17 3888 3887 3945 3887
q18 9655 9287 9437 9287
q19 1894 1775 1795 1775
q20 2393 2054 2053 2053
q21 8783 8645 8706 8645
q22 927 845 833 833
Total cold run time: 81904 ms
Total hot run time: 79119 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808650336
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 0a8517e000a30198f189dea495df2f2d1c9014b9, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5279 5168 5140 5140
q2 359 165 156 156
q3 2047 2004 2028 2004
q4 1410 1363 1379 1363
q5 3951 3954 3986 3954
q6 258 132 133 132
q7 1425 879 878 878
q8 2772 2760 2753 2753
q9 9542 9543 9477 9477
q10 3445 3490 3535 3490
q11 387 266 257 257
q12 435 281 288 281
q13 4578 4128 4158 4128
q14 330 290 292 290
q15 642 569 548 548
q16 675 607 589 589
q17 1121 1110 1104 1104
q18 7954 7512 7555 7512
q19 1670 1674 1663 1663
q20 533 334 307 307
q21 4646 4361 4323 4323
q22 524 413 405 405
Total cold run time: 53983 ms
Total hot run time: 50754 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5019 5064 5063 5063
q2 330 223 269 223
q3 4067 4028 3970 3970
q4 2793 2771 2783 2771
q5 9531 9579 9567 9567
q6 244 125 124 124
q7 2662 2256 2383 2256
q8 4818 4814 4862 4814
q9 12997 12905 13077 12905
q10 4061 4125 4136 4125
q11 780 651 633 633
q12 1022 853 820 820
q13 4291 3855 3855 3855
q14 386 342 359 342
q15 594 541 542 541
q16 772 694 691 691
q17 3870 3909 3933 3909
q18 9676 9465 9492 9465
q19 1748 1772 1766 1766
q20 2405 2060 2022 2022
q21 8690 8660 8810 8660
q22 909 827 872 827
Total cold run time: 81665 ms
Total hot run time: 79349 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #26904:
URL: https://github.com/apache/doris/pull/26904#discussion_r1391111207
##########
be/src/runtime/thread_context.cpp:
##########
@@ -23,34 +23,27 @@
namespace doris {
class MemTracker;
-DEFINE_STATIC_THREAD_LOCAL(ThreadContext, ThreadContextPtr, _ptr);
-
-ThreadContextPtr::ThreadContextPtr() {
- INIT_STATIC_THREAD_LOCAL(ThreadContext, _ptr);
- init = true;
-}
-
AttachTask::AttachTask(const std::shared_ptr<MemTrackerLimiter>& mem_tracker,
const TUniqueId& task_id, const TUniqueId& fragment_instance_id) {
- SwitchBthreadLocal::switch_to_bthread_local();
+ ThreadLocalHandle::handle_thread_local();
Review Comment:
rename create_thread_local_if_not_exits()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808868477
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.52 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17163747525 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #26904:
URL: https://github.com/apache/doris/pull/26904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808630863
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on code in PR #26904:
URL: https://github.com/apache/doris/pull/26904#discussion_r1391184250
##########
be/src/runtime/thread_context.cpp:
##########
@@ -23,34 +23,27 @@
namespace doris {
class MemTracker;
-DEFINE_STATIC_THREAD_LOCAL(ThreadContext, ThreadContextPtr, _ptr);
-
-ThreadContextPtr::ThreadContextPtr() {
- INIT_STATIC_THREAD_LOCAL(ThreadContext, _ptr);
- init = true;
-}
-
AttachTask::AttachTask(const std::shared_ptr<MemTrackerLimiter>& mem_tracker,
const TUniqueId& task_id, const TUniqueId& fragment_instance_id) {
- SwitchBthreadLocal::switch_to_bthread_local();
+ ThreadLocalHandle::handle_thread_local();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1809421624
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808490693
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on code in PR #26904:
URL: https://github.com/apache/doris/pull/26904#discussion_r1391108653
##########
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##########
@@ -161,9 +163,9 @@ inline void ThreadMemTrackerMgr::pop_consumer_tracker() {
_consumer_tracker_stack.pop_back();
}
-inline void ThreadMemTrackerMgr::consume(int64_t size, bool large_memory_check) {
+inline void ThreadMemTrackerMgr::consume(int64_t size, int skip_large_memory_check) {
Review Comment:
Because `int` can support nesting, Maybe there might be a better way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26904:
URL: https://github.com/apache/doris/pull/26904#discussion_r1391051000
##########
be/src/runtime/thread_context.h:
##########
@@ -101,63 +113,26 @@ class RuntimeState;
extern bthread_key_t btls_key;
-// Using gcc11 compiles thread_local variable on lower versions of GLIBC will report an error,
-// see https://github.com/apache/doris/pull/7911
-//
-// If we want to avoid this error,
-// 1. For non-trivial variables in thread_local, such as std::string, you need to store them as pointers to
-// ensure that thread_local is trivial, these non-trivial pointers will uniformly call destructors elsewhere.
-// 2. The default destructor of the thread_local variable cannot be overridden.
-//
-// This is difficult to implement. Because the destructor is not overwritten, it means that the outside cannot
-// be notified when the thread terminates, and the non-trivial pointers in thread_local cannot be released in time.
-// The func provided by pthread and std::thread doesn't help either.
-//
-// So, kudu Class-scoped static thread local implementation was introduced. Solve the above problem by
-// Thread-scoped thread local + Class-scoped thread local.
-//
-// This may look very trick, but it's the best way I can find.
-//
-// refer to:
-// https://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Thread-Local.html
-// https://stackoverflow.com/questions/12049684/
-// https://sourceware.org/glibc/wiki/Destructor%20support%20for%20thread_local%20variables
-// https://www.jianshu.com/p/756240e837dd
-// https://man7.org/linux/man-pages/man3/pthread_tryjoin_np.3.html
-class ThreadContextPtr {
-public:
- ThreadContextPtr();
- // Cannot add destructor `~ThreadContextPtr`, otherwise it will no longer be of type POD, the reason is as above.
-
- // TCMalloc hook is triggered during ThreadContext construction, which may lead to deadlock.
- bool init = false;
-
- DECLARE_STATIC_THREAD_LOCAL(ThreadContext, _ptr);
-};
-
-inline thread_local ThreadContextPtr thread_context_ptr;
-
-// To avoid performance problems caused by frequently calling `bthread_getspecific` to obtain bthread TLS
-// in tcmalloc hook, cache the key and value of bthread TLS in pthread TLS.
-inline thread_local ThreadContext* bthread_context;
-inline thread_local bthread_t bthread_id;
+// Is true after ThreadContext construction.
+inline thread_local bool pthread_context_ptr_init = false;
+inline thread_local constinit ThreadContext* thread_context_ptr;
+// use mem hook to consume thread mem tracker.
+inline thread_local bool use_mem_hook = false;
// The thread context saves some info about a working thread.
// 2 required info:
// 1. thread_id: Current thread id, Auto generated.
-// 2. type: The type is a enum value indicating which type of task current thread is running.
+// 2. type(abolished): The type is a enum value indicating which type of task current thread is running.
// For example: QUERY, LOAD, COMPACTION, ...
// 3. task id: A unique id to identify this task. maybe query id, load job id, etc.
+// 4. ThreadMemTrackerMgr
//
// There may be other optional info to be added later.
class ThreadContext {
public:
- ThreadContext() {
- thread_mem_tracker_mgr.reset(new ThreadMemTrackerMgr());
- if (ExecEnv::ready()) thread_mem_tracker_mgr->init();
- }
+ ThreadContext() { thread_mem_tracker_mgr = std::make_unique<ThreadMemTrackerMgr>(); }
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```cpp
ThreadContext() { thread_mem_tracker_mgr = std::make_unique<ThreadMemTrackerMgr>(); }
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808776051
TeamCity be ut coverage result:
Function Coverage: 36.63% (8375/22861)
Line Coverage: 29.21% (67986/232710)
Region Coverage: 27.81% (35119/126271)
Branch Coverage: 24.65% (17962/72868)
Coverage Report: http://coverage.selectdb-in.cc/coverage/47284f0aaadada152de7523ed30f37d44b654a94_47284f0aaadada152de7523ed30f37d44b654a94/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1809421573
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](tls) Manually track memory in Allocator instead of mem hook and ThreadContext life cycle to manual control [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26904:
URL: https://github.com/apache/doris/pull/26904#issuecomment-1808653885
TeamCity be ut coverage result:
Function Coverage: 36.63% (8375/22861)
Line Coverage: 29.21% (67978/232710)
Region Coverage: 27.81% (35112/126271)
Branch Coverage: 24.65% (17961/72868)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a8517e000a30198f189dea495df2f2d1c9014b9_0a8517e000a30198f189dea495df2f2d1c9014b9/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org