[GitHub] [incubator-doris] morningman opened a new pull request #4885: [Optimize] Send fragments in batch to reduce RPC

[GitBox <gi...@apache.org>]

morningman opened a new pull request #4885:
URL: https://github.com/apache/incubator-doris/pull/4885


   ## Proposed changes
   
   This CL mainly changes:
   
   1. Send and execute Fragments in batch
   
       In the previous implementation, a SQL execution plan may generate multiple Fragments.
   Even if these Fragments are sent to the same BE node, they are sent separately in multiple RPCs.
   These Fragments will have some common information, such as DescriptorTable, QueryGlobals, etc.
   Sending one by one through RPC will repeatedly DeSer these information, resulting in unnecessary overhead.
   
       After the modification, the Coordinator merges multiple Fragments of same BE and sends them in one RPC,
   and extracts common information to avoid multiple repeated DeSer operations. At the same time,
   the Fragment execution part on the BE side is modified, multiple Fragments are executed in one request,
   and the common structure is avoided constructing repeatedly.
   
       In the local single-node test, a SQL which generates 300 fragments, the execution time is reduced from
   7 seconds to 2 seconds. The main reduction part is the receiving and executing Fragment part on the BE side.
   The remaining time is mainly spent on query planning on the FE side, which will be improved in next PR.
   
   2. Add the time-consuming part of FE logic in Profile
   
       Including SQL analysis, planning, Fragment scheduling and sending on the FE side, and the time to fetch data.
   
   ## Types of changes
   
   - [x] Code refactor (Modify the code structure, format the code, etc...)
   
   ## Checklist
   
   - [x] I have create an issue on (Fix #4656), and have described the bug/feature there in detail
   
   ## Further comments
   
   The query planner is still time-consuming when processing complex SQL. It will be optimized later.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [incubator-doris] morningman closed pull request #4885: [Optimize] Send fragments in batch to reduce RPC

[GitBox <gi...@apache.org>]

morningman closed pull request #4885:
URL: https://github.com/apache/incubator-doris/pull/4885


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org