You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/07/19 08:48:22 UTC
[GitHub] [doris] luwei16 opened a new pull request, #11006: (cold_on_s3) Isolate local and remote queries using different scanner…
luwei16 opened a new pull request, #11006:
URL: https://github.com/apache/doris/pull/11006
… thread pools (#276)
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] luwei16 commented on a diff in pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
luwei16 commented on code in PR #11006:
URL: https://github.com/apache/doris/pull/11006#discussion_r929723974
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
Review Comment:
changed to 16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] luwei16 commented on a diff in pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
luwei16 commented on code in PR #11006:
URL: https://github.com/apache/doris/pull/11006#discussion_r929724579
##########
be/src/exec/olap_scan_node.cpp:
##########
@@ -1503,6 +1503,7 @@ void OlapScanNode::transfer_thread(RuntimeState* state) {
* 4. Regularly increase the priority of the remaining tasks in the queue to avoid starvation for large queries
*********************************/
PriorityThreadPool* thread_pool = state->exec_env()->scan_thread_pool();
+ PriorityThreadPool* s3_thread_pool = state->exec_env()->s3_scan_thread_pool();
Review Comment:
done
##########
be/src/runtime/exec_env.h:
##########
@@ -200,6 +201,7 @@ class ExecEnv {
// TODO(cmy): find a better way to unify these 2 pools.
PriorityThreadPool* _scan_thread_pool = nullptr;
+ PriorityThreadPool* _s3_scan_thread_pool = nullptr;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] luwei16 commented on a diff in pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
luwei16 commented on code in PR #11006:
URL: https://github.com/apache/doris/pull/11006#discussion_r929724365
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
+// number of s3 scanner thread pool queue size
+CONF_Int32(doris_s3_scanner_thread_pool_queue_size, "102400");
Review Comment:
changed to 10240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #11006:
URL: https://github.com/apache/doris/pull/11006#issuecomment-1195433911
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
dataroaring merged PR #11006:
URL: https://github.com/apache/doris/pull/11006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] pengxiangyu commented on a diff in pull request #11006: (performance)[scanner] Isolate local and remote queries using different scanner…
Posted by GitBox <gi...@apache.org>.
pengxiangyu commented on code in PR #11006:
URL: https://github.com/apache/doris/pull/11006#discussion_r926229647
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
+// number of s3 scanner thread pool queue size
+CONF_Int32(doris_s3_scanner_thread_pool_queue_size, "102400");
Review Comment:
102400 is too big,
##########
be/src/runtime/exec_env.h:
##########
@@ -200,6 +201,7 @@ class ExecEnv {
// TODO(cmy): find a better way to unify these 2 pools.
PriorityThreadPool* _scan_thread_pool = nullptr;
+ PriorityThreadPool* _s3_scan_thread_pool = nullptr;
Review Comment:
_s3_scan_thread_pool need to be _remote_scan_thread_pool
##########
be/src/exec/olap_scan_node.cpp:
##########
@@ -1503,6 +1503,7 @@ void OlapScanNode::transfer_thread(RuntimeState* state) {
* 4. Regularly increase the priority of the remaining tasks in the queue to avoid starvation for large queries
*********************************/
PriorityThreadPool* thread_pool = state->exec_env()->scan_thread_pool();
+ PriorityThreadPool* s3_thread_pool = state->exec_env()->s3_scan_thread_pool();
Review Comment:
Use remote_thread_pool, it is not only for S3
##########
be/src/common/config.h:
##########
@@ -798,6 +798,11 @@ CONF_Int32(s3_transfer_executor_pool_size, "2");
CONF_Bool(enable_time_lut, "true");
+// number of s3 scanner thread pool size
+CONF_Int32(doris_s3_scanner_thread_pool_thread_num, "48");
Review Comment:
DEFAULT VALUE 48 is too big, it will cost too many cpu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org