You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/05/27 04:14:58 UTC
[GitHub] [incubator-doris] caiconghui opened a new issue #3696: [Proposal] Change some default config in be to make doris more friendly to new user and make brpc_num_threads configurable to get better performance for qps Support
caiconghui opened a new issue #3696:
URL: https://github.com/apache/incubator-doris/issues/3696
Now, there are many rpcs between be and fe, and some default config in be may cause poor performance for query or stream load. In our experiment, we find that when we execute stream load in doris, the txn can be easily publish timeout and query can be easily rpc timeout, http request to be can be easily no response.
And we read the code for server in be, and we find the num of thrift server threads is too small to handle so many rpc which include exec_plan_fragment and submit_tasks at the same time, and http server use reactor model to process request, when the stream load performance is poor (server thread would be blocked in rpc or other blocked function) and the num of threads in http server is small, it will cause http server can not handle http request in time.
Finally, we do some experiments after change the some config in be, and we find that query and stream load both get good performance and http server can response in time.
we use thread pool in java client to make 10000 stream load requests to fe with 5 be instance. Every request has one row data, and we have 500 tables in on db, every table will be inserted 20 rows. the total time cost for java client is 35398ms, and some stream load result are as following:
{
"TxnId": 276003,
"Label": "86292a00-511b-47de-82e2-6dfa9c2d219d",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 12,
"LoadTimeMs": 31,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 15
}
{
"TxnId": 276006,
"Label": "7d6023a4-5fb5-4551-9327-4c758f231851",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 12,
"LoadTimeMs": 31,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 15
}
{
"TxnId": 275256,
"Label": "3b29cd47-2aa8-4255-9d37-df6148570e93",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 11,
"LoadTimeMs": 34,
"BeginTxnMs": 0,
"ReadBufferMs": 0,
"WriteBufferMs": 12,
"StreamLoadPutMs": 0,
"CommitAndPublishMs": 18
}
load all time 35398
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3696: [Proposal] Change some default config in be to make doris more friendly to new user and make brpc_num_threads configurable to get better performance for qps
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3696:
URL: https://github.com/apache/incubator-doris/issues/3696#issuecomment-634457445
Looking forward to your PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui closed issue #3696: [Proposal] Change some default config in be to make doris more friendly to new user and make brpc_num_threads configurable to get better performance for qps
Posted by GitBox <gi...@apache.org>.
caiconghui closed issue #3696:
URL: https://github.com/apache/incubator-doris/issues/3696
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org