You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/05 17:25:01 UTC
[GitHub] [doris] k-i-d-d opened a new pull request, #10629: [Enhancement] Improve TCMalloc Hook consume MemTracker performance
k-i-d-d opened a new pull request, #10629:
URL: https://github.com/apache/doris/pull/10629
# Proposed changes
Issue Number: close #10583
## Problem Summary:
#### Motivation
In the previous test, consume mem tracker based on tcmalloc hook is expected to bring 2% performance loss to the query.
But in extreme cases, when load JSON data, the tcmalloc hook may bring about a 30% performance loss.
The performance loss is divided into five parts:
1. Calculate the actual system memory size of this alloc/free, loss 3%;
2. After accumulating to 2M, consume mem tracker, loss 2%;
3. Get bthread tls, loss 7%;
4. The function call introduced for consuming mem tracker in tls;
5. The hook itself;
#### Solution
1. Avoid trying to get bthread tls in every hook;
2. Optimize unnecessary conditional judgments;
3. Reduce the number of function call layers in the hook;
4. Modified to consume mem tracker after accumulating 4M;
The optimized performance loss is divided into three parts:
1. Calculate the actual system memory size of this alloc/free;
2. Consume mem tracker after accumulating 4M;
3. The hook itself;
In the test environment, this can reduce the performance loss of tcmalloc hook from 6% to 2%.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10629: [Enhancement] Improve TCMalloc Hook consume MemTracker performance
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #10629:
URL: https://github.com/apache/doris/pull/10629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] kpfly commented on pull request #10629: [Enhancement] Improve TCMalloc Hook consume MemTracker performance
Posted by GitBox <gi...@apache.org>.
kpfly commented on PR #10629:
URL: https://github.com/apache/doris/pull/10629#issuecomment-1178764456
before this patch，when load JSON data, the tcmalloc hook may bring about a 30% performance loss.
how about after this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] kpfly commented on pull request #10629: [Enhancement] Improve TCMalloc Hook consume MemTracker performance
Posted by GitBox <gi...@apache.org>.
kpfly commented on PR #10629:
URL: https://github.com/apache/doris/pull/10629#issuecomment-1178775502
>
nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] k-i-d-d commented on pull request #10629: [Enhancement] Improve TCMalloc Hook consume MemTracker performance
Posted by GitBox <gi...@apache.org>.
k-i-d-d commented on PR #10629:
URL: https://github.com/apache/doris/pull/10629#issuecomment-1178772483
> before this patch，when load JSON data, the tcmalloc hook may bring about a 30% performance loss. how about after this patch?
10%,
In addition to load large JSON, other load are usually only have a loss of less than 2%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org