You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/01/07 09:32:58 UTC
[GitHub] [incubator-doris] caiconghui opened a new issue #5204: [Broker Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full
caiconghui opened a new issue #5204:
URL: https://github.com/apache/incubator-doris/issues/5204
Now, many user use broker load to load hdfs data into doris, and they will submit many broker job at one time, like #5150 and loading task usually takes long time to finish task while pending task cost little time to finish task.
when task queue in loadingLoadTaskScheduler is full, there is no need to begin txn for a new load job, which will just wait for
other broker job to finish their loading task. Once start a new transaction, the broker job's status will be transformed from pending to loading, which may be confused for user that it is not load slowly but for the limited thread resource.
So I will make two change,
1. split async_load_task_pool_size config into two configs: async_pending_load_task_pool_size, async_loading_load_task_pool_size
2. check the task queue in loadingLoadTaskScheduler is full, once full, we should not begin transaction immediately for bulk load job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman closed issue #5204: [Broker Load] Load job should not begin transaction when task queue in loadingLoadTaskScheduler is full
Posted by GitBox <gi...@apache.org>.
morningman closed issue #5204:
URL: https://github.com/apache/incubator-doris/issues/5204
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org