[jira] [Resolved] (CALCITE-4494) Improve planning performance with RelSubset check for Rel presence

["Stamatis Zampetakis (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CALCITE-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stamatis Zampetakis resolved CALCITE-4494.
------------------------------------------
    Resolution: Fixed

Fixed in [78cc3e3f739704ec4fa36bee9b465b61a672f873|https://github.com/apache/calcite/commit/78cc3e3f739704ec4fa36bee9b465b61a672f873]. Thanks for making the planner faster [~aigor]!

>  Improve planning performance with RelSubset check for Rel presence
> -------------------------------------------------------------------
>
>                 Key: CALCITE-4494
>                 URL: https://issues.apache.org/jira/browse/CALCITE-4494
>             Project: Calcite
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 1.26.0
>         Environment: All environments
>            Reporter: Igor Lozynskyi
>            Assignee: Igor Lozynskyi
>            Priority: Major
>              Labels: performance, pull-request-available
>             Fix For: 1.27.0
>
>         Attachments: CalcitePerf_Planning_RelList_consumes_a_lot.png, CalcitePerf_Planning_TPCH_Q7_RelList_consumes_a_lot.png, CalcitePerf_Planning_after_improvements.png
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Problem*
> Currently, the planning process shows a performance degradation when comparing to version 1.25. Worse palling time seems to affect most queries, but it is especially clear for queries with many Rel nodes (especially with multiple joins).
> In a downstream project, we have a stress test that checks the planning time. In some cases, the planning time is increased by x4 (for a query with 28 joins).
> The main contributing factor (but not the only one) for the slow-down could be traced to [https://github.com/apache/calcite/pull/2222/files].
> *Potential Solution*
> As it was mentioned by the reviewers, we may improve the current situation with some tiny changes:
>  * Introduce a method to check that a Rel node belongs to the RelSubset instead of getting all Rel nodes (the current code may take up to 60% of the planning time).
> *  Improve the null check in RelMdPredicates by building an error message in RelMdPredicates.ExprsItr only when it is required (may additionally take 10% of the planning time due to SortedMap.toString() being expensive when frequently called).
> With these 2 changes, I was able to regain most of the lost planning performance.
> The following flame graph clearly shows that the call to RelSubset.getRelList() from VolcanoRuleCall.onMatch() is expensive (28 join query):
>  !CalcitePerf_Planning_RelList_consumes_a_lot.png|width=865,height=365!
> After the proposed improvements, the flame graph shows the following (28 join query):
>     !CalcitePerf_Planning_after_improvements.png|width=561,height=472!
> It is clear that the HintStrategyTable.isRuleExcluded() call is expensive, but the overall picture is much better.
> Also, in my environment, the TPC-H Q7 test takes ~20% less time (39.6 sec vs 32.9 sec) after the proposed improvements. Here, the flame graph also shows that ordinary queries are also affected by the redundant RelSubset.getRelList() calls:
> !CalcitePerf_Planning_TPCH_Q7_RelList_consumes_a_lot.png|width=856,height=573!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)