You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Aleksey Plekhanov (Jira)" <ji...@apache.org> on 2022/02/03 07:44:00 UTC

[jira] [Updated] (IGNITE-16315) Calcite engine. Query start request contains a lot of data

     [ https://issues.apache.org/jira/browse/IGNITE-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aleksey Plekhanov updated IGNITE-16315:
---------------------------------------
    Labels: calcite3-required  (was: calcite2-required calcite3-required)

> Calcite engine. Query start request contains a lot of data
> ----------------------------------------------------------
>
>                 Key: IGNITE-16315
>                 URL: https://issues.apache.org/jira/browse/IGNITE-16315
>             Project: Ignite
>          Issue Type: Improvement
>            Reporter: Aleksey Plekhanov
>            Assignee: Aleksey Plekhanov
>            Priority: Major
>              Labels: calcite3-required
>          Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> For simple queries SQL engine most of the time spend in writing/reading query start requests, which contains a lot of data. Nested instances of {{ColocationGroup}} class contain assignments for each partition ({{{}List<List<UUID>>{}}}). Transferred size can be reduced if we compact assignments somehow. The target colocation group from fragment description contains redundant synthetic partitions, this also can be optimized.
> Messages workflow is not optimal too. First, we send {{QueryStartRequest}} to the remote nodes, remotes reply with the QueryStartResponse messages. After that remotes send batches with data to the target nodes and receive acks for each batch (acks required to limit inbox workload). When query execution is finished, the node initiator sends {{QueryCloseMessage}} to the remote nodes, remotes close queries, and sends back {{ErrorMessage}} to the initiator with the {{ExecutionCancelledException}} error (which is ignored on the initiator node).  
> Also, some other optimizations are possible. Proposed changes:
>  * Implement compaction of assignments of {{ColocationGroup}}
>  * Reduce target colocation group partitions count
>  * Fix caching of query plans (store original SQL as key, not parsed SQL, to avoid redundant parsing)
>  * Change messages workflow (don't send ack messages for the last batch since it is redundant, self-close remote queries, and don't send close query messages to remote nodes, if we know for sure that it's already self-closed, don't send query start response if we already have sent batch for the same fragment before)
>  * Reduce count of {{RexBuilder}} creation on the execution phase (RexBuilder is stateless and can be used one static instance)
>  * Reduce count of Calcite types creation on the execution phase



--
This message was sent by Atlassian Jira
(v8.20.1#820001)