You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/21 14:06:13 UTC
[GitHub] [incubator-doris] caiconghui opened a new issue #3368: [Proposal] Making doris stream load more rubust with higher concurrent performance
caiconghui opened a new issue #3368:
URL: https://github.com/apache/incubator-doris/issues/3368
The main purpose of this proposal is to make doris stream load more rubust with higher concurrent peformance.
Now, in our production environment, we mainly use stream load to load business data into doris.
And we found the following problem:
1. the transaction processing in GlobalTransactionMgr performance is poor, because the transaction lock is global without any isolation, for some operation we need to iterate the whole map.
2. Fe thrift server use Executors.newFixedThreadPool and Executors.newCachedThreadPool to construct thread pool, which may cause oom or too many thread to be created and then crash
3. the rpc cache client num is not limited in be and when rpc client timeout, the request task is still blocked in fe, the client encounter rpc_timeout and retry, it may cause disaster for fe sometimes.
4. the tablet writer open cost too much time, because when user doesn't specify the partition, the all tablet writer would be opened.
Implementation
So the following work we need to implement:
1. Support db level isolation for transaction processing and use ArrayDeque to stored the finished transaction
2. Remove all code that used newFixedThreadPool or newCachedThreadPool, construct the thread pool explicitly, support thread pool metric monitor
3. use client pool to manage rpc client, set the core and max cache client num, use try lock instead of lock to support server task be can be canceled in time due to timeout
4. open the tablet writer when needed not open all when partitions of table are not specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3368: [Proposal] Making doris stream load more robust with higher concurrent performance
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3368:
URL: https://github.com/apache/incubator-doris/issues/3368#issuecomment-617497647
@caiconghui Great job!
Could we support table level isolation for transaction processing？ If we only support DB level isolation and one DB has hundreds of tables, the transaction performance will still poor!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3368: [Proposal] Making doris stream load more robust with higher concurrent performance
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3368:
URL: https://github.com/apache/incubator-doris/issues/3368#issuecomment-617527413
> @kangkaisen I think if we want to support table level isolation, there is still many work to do, because now the meta design for doris is on db level isolation, and this PR not only support db level isolation, but also do some optimization for txn manage task, the transaction performance may not be poor although one DB has hundreds of tables, we still need some performance data to decide to if we need to support table level isolation, we just do it now.
OK, When you do all work. would better give a benchmark like https://github.com/apache/incubator-doris/issues/2377. Thanks you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #3368: [Proposal] Making doris stream load more robust with higher concurrent performance
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #3368:
URL: https://github.com/apache/incubator-doris/issues/3368#issuecomment-617512726
@kangkaisen I think if we want to support table level isolation, there is still many work to do, because now the meta design for doris is on db level isolation, and this PR not only support db level isolation, but also do some optimization for txn manage task, the transaction performance may not be poor although one DB has hundreds of tables, we still need some performance data to decide to if we need to support table level isolation, we just do it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org