You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/29 01:58:50 UTC

[GitHub] [incubator-doris] WingsGo opened a new issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

WingsGo opened a new issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966


   In production enviorment, the data loading is 7*24 with lots of txns, but we found that the TransactionMgr costs too much memeory, which will leads OOM.If we want to improve the concurrent of stream load, we must optimize the memory usage in order to support frequent data load.There are some way such as clean up useless data after txn finished #3130, reduce the interval of clean txn, use ZGC to recycle memory in a short time....But It is not enough, I found FE will OOM accidental cause by TransactionMgr usage.
   There are two way to reduce the memory usage, one is change the data struct in DatabaseTransactionMgr to save memeory, the other is scattered the Txn to other follower FE to share the memory usage, I will invetigate the feasibility of two solutions, any advice or solution is welcome.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] wangbo commented on issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

Posted by GitBox <gi...@apache.org>.
wangbo commented on issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966#issuecomment-751681522


   I think the ```Check Point Thread``` can run in follower.
   It's not necessary for ```Master FE``` to run it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] WingsGo commented on issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

Posted by GitBox <gi...@apache.org>.
WingsGo commented on issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966#issuecomment-653489222


   I found that most of memory usage in DatabaseTransactionMgr is a HashMap struct idToFinalStatusTransactionState, following shows that 270W TransactionState object almost cost 50G memory, but some data in TransactionState is useless, so we have no need to save the data.
   
   ```
   Label | Number of Objects | Used Heap Size | Retained Heap Size
   -- | -- | -- | --
   java.util.HashMap$NodeFirst 10 of 2,718,787 objects | 2,718,787 | 130,501,776 | 53,717,135,376
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] WingsGo edited a comment on issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966#issuecomment-650864550


   By the way, too much DatabaseTransactionMgr will cause some more serious accident.For example, if DatabaseTransactionMgr costs too much memory, checkpoint image will not work because it will check java heap memory usage in order to create image in memory, so when FE down cause by OOM, it will replay too much EditLog, in the worst case, it will replay more than one hour, It means that data load is not available in the meantime.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] WingsGo commented on issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

Posted by GitBox <gi...@apache.org>.
WingsGo commented on issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966#issuecomment-650864550


   By the way, too much DatabaseTransactionMgr will cause some more serious accident.For example, if DatabaseTransactionMgr costs too much memory, checkpoint image will not work because it will check java heap memory in order to create image in memory, so when FE down cause by OOM, it will replay too much EditLog, in the worst case, it will replay more than one hour, It means that data load is not available in the meantime.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] WingsGo edited a comment on issue #3966: [Proposal]Optimize the memory consumption of TransactionMgr

Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3966:
URL: https://github.com/apache/incubator-doris/issues/3966#issuecomment-653489222


   I found that most of memory usage in DatabaseTransactionMgr is a HashMap struct idToFinalStatusTransactionState, following shows that 270W TransactionState object almost cost 50G memory, but some data in TransactionState is useless, so we have no need to save the data. So I will remove unnessary data in TransactionState in order to avoid too much memeory usage and too long time editlog replay.
   
   ```
   Label | Number of Objects | Used Heap Size | Retained Heap Size
   -- | -- | -- | --
   java.util.HashMap$NodeFirst 10 of 2,718,787 objects | 2,718,787 | 130,501,776 | 53,717,135,376
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org