[jira] [Commented] (CALCITE-2970) Performance issue when enabling abstract converter for EnumerableConvention

["Xiening Dai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961619#comment-16961619 ] 

Xiening Dai commented on CALCITE-2970:
--------------------------------------

I spent some time debugging this issue. I think I know what is going on. 

When useAbstractConvertersForConversion is set to true, planner inserts an AbstractConverter node to convert input into the collation trait that's required by SortMergedJoin. And during the ExpandConversionRule, the planner calls into RelCollationTraitDef.convert() and adds an LogicalSort to the input. The new LogicalSort, combined with other existing logical nodes, significantly expand the search space as they trigger a large number of rules, such as ProjectMergeRule, AggregateProjectMergeRule, etc. These rules produce too many alternatives which then would trigger more rules. So the optimization latency spikes (from 3s to 52s).

I did a quick experiment to create EnumerableSort, instead of LogicalSort, within RelCollationDef.convert(). The planning time is down from 52s to 9s. And join 6 way can be completed with 12s. See - https://github.com/xndai/calcite/tree/calcite-2970

I understand we cannot just create EnumerableSort as there maybe other conventions. But I think it's totally reasonable to just create a corresponding physical sort when convention is known. When we know exactly the required convention and collation trait, there's no reason to create yet another logical node. 

In order to achieve that, the framework would need to provide a way to create a physical node with given convention not through user rules. I don't have a clear solution in my mind, and I would like to hear your comments [~hyuan] [~zabetak] [~julianhyde]. I think this will be an important enhancement to the calcite framework.

> Performance issue when enabling abstract converter for EnumerableConvention
> ---------------------------------------------------------------------------
>
>                 Key: CALCITE-2970
>                 URL: https://issues.apache.org/jira/browse/CALCITE-2970
>             Project: Calcite
>          Issue Type: Bug
>          Components: core
>            Reporter: Haisheng Yuan
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> If we enable the use of abstract converter for {{EnumerableConvention}}, by making {{useAbstractConvertersForConversion}} return true, {{JDBCTest.testJoinManyWay}} will not complete. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)