You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/02/13 13:34:57 UTC
[GitHub] [incubator-doris] chaoyli opened a new issue #2896: Import a new
ThreadPool Framework
chaoyli opened a new issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2896: Import a
new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-586089869
1. import scoped_refptr from gutil #2899
2. import ConditionVariable from Kudu #2905
3. import ThreadPool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2896: Import a
new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-585758880
This ThreadPool have three features.
1. Using a token mechanism to receive tasks.
In Doris, tasks are submitted into ThreadPool without group.
If a task in one group should be handled serially, it will stall all tasks in ThreadPool.
For example, MemTableFlush tasks belong to different tablets can be flushed concurrently
and tasks belongs one tablet should be flushed serially.
So we need a token mechanism to divide tasks into groups.
2. Support set min_threads and max_threads in ThreadPool.
Now, Doris will start all threads in ThreadPool. It is not efficiently.
We should start threads as little as possible if have no tasks.
At peek time, ThreadPool can expand threads up to max_threads.
3. Better trace to resource usage of any thread than C++ library.
All threads in ThreadPool are in black box. Resource usage in this ThreadPool is not clear.
New thread implementation should handle this question.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2896: Import a new
ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-586089869
1. import scoped_refptr from gutil #2899
2. import ConditionVariable from Kudu
3. import ThreadPool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2896: Import a
new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-586089869
1. import scoped_refptr from gutil #2899 [done]
2. import ConditionVariable from Kudu #2905
3. import ThreadPool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2896: Import a
new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-586089869
1. import scoped_refptr from gutil #2899
2. import ConditionVariable from Kudu 2905
3. import ThreadPool
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli closed issue #2896: Import a new
ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli closed issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #2896: Import a
new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-586089869
1. import scoped_refptr from gutil #2899 [done]
2. import ConditionVariable from Kudu #2905
3. import ThreadPool #2915
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2896: Import a new
ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-585758880
This ThreadPool have three features.
**Using a token mechanism to receive tasks.**
In Doris, tasks are submitted into ThreadPool without group.
If a task in one group should be handled serially, it will stall all tasks in ThreadPool.
For example, MemTableFlush tasks belong to different tablets can be flushed concurrently
and tasks belongs one tablet should be flushed serially.
2. Support set min_threads and max_threads in ThreadPool.
At peek time, ThreadPool can expand threads up to max_threads.
3. New thread implementation which is better to trace on resource usage
than C++ library.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2896: Import a new
ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-585757484
Now, Doris have multiple ThreadPool in different code path.
And all of them have a little difference.
So I import a new ThreadPool to unify all of these.
And also to further use.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli removed a comment on issue #2896: Import
a new ThreadPool Framework
Posted by GitBox <gi...@apache.org>.
chaoyli removed a comment on issue #2896: Import a new ThreadPool Framework
URL: https://github.com/apache/incubator-doris/issues/2896#issuecomment-585757484
Now, Doris have multiple ThreadPool in different code path.
And all of them have a little difference.
So I import a new ThreadPool to unify all of these.
And also to further use.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org