You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/21 15:52:14 UTC
[GitHub] [incubator-doris] morningman opened a new issue #5280: [Performance] Import the load performace
morningman opened a new issue #5280:
URL: https://github.com/apache/incubator-doris/issues/5280
Doris' load performance is very bad in some cases. Especially when using `broker load` to load large quantities of data, the system resources may be idle, but the load is very slow.
Currently I found the following problems:
1. The execution plan of Broker Load has only one concurrency on a single node, so even if the system resources are idle, the load speed cannot be improved by increasing the concurrency.
2. Memtable is written to disk serially. By default, a disk has two flush threads, and the current flush thread pool strategy is serial execution. In fact, each memtable can be written to disk in parallel.
3. A loadChannel may correspond to data sent by multiple OlapTableSinks, and because of the problem of LoadChannel lock granularity, the data receiving is a serial operation. Therefore, in some cases, even if the BE node is added, the load speed cannot be improved.
I am trying to solve these problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #5280: [Performance] Import the load performace
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #5280:
URL: https://github.com/apache/incubator-doris/issues/5280#issuecomment-766830342
duplicate #5281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #5280: [Performance] Import the load performace
Posted by GitBox <gi...@apache.org>.
morningman closed issue #5280:
URL: https://github.com/apache/incubator-doris/issues/5280
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #5280: [Performance] Import the load performace
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #5280:
URL: https://github.com/apache/incubator-doris/issues/5280#issuecomment-766830342
duplicate #5281
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #5280: [Performance] Import the load performace
Posted by GitBox <gi...@apache.org>.
morningman closed issue #5280:
URL: https://github.com/apache/incubator-doris/issues/5280
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org