[jira] [Comment Edited] (CALCITE-5365) RelMdExpressionLineage#createAllPossibleExpressions will create many RexCall lead to OOM

["Arnaud Jegou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CALCITE-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128786#comment-18128786 ] 

Arnaud Jegou edited comment on CALCITE-5365 at 10/19/26 2:23 PM:
-----------------------------------------------------------------

_edit: reframed_

Splitting the condition makes sense to me. Part of the blowup comes from something else though: above a union, lineage combines fields from different union inputs, which can never happen (CALCITE-7869). Here only 3 x 3 = 9 of the 3^14 combinations are valid. Fixing that keeps the result small in most cases, and splitting still helps on top of it when the conjuncts relate different unions. A limit still makes sense as a safeguard for plans that really have many valid combinations.

note: the query in the description uses left joins, and {{getAllPredicates}} returns null for an outer join, so as written it does not reach {{createAllPossibleExpressions}}. It does with inner joins and {{getAllPredicates}} does build the 3^14 combinations




was (Author: JIRAUSER310786):
The query in the description uses left joins, and `getAllPredicates` returns null for an outer join, in 1.30.0 as on main, so as written it does not reach `createAllPossibleExpressions`. With inner joins it does: `getAllPredicates` builds 3^14 combinations for the seven-key join condition.

Most of those combinations cannot occur, which is now tracked in CALCITE-7869: above a Union, lineage combines fields from different inputs. A row of the join pairs one input of t1 with one input of t2, so only 3 x 3 = 9 combinations are valid. On main, the same shape with two keys instead of seven returns 81 predicates instead of 9.

Fixing CALCITE-7869 would also address the concern above that a limit loses predicate information: the valid combinations would stay small, so a limit would only apply to plans that really have many source combinations, such as joins of several wide unions. Splitting the condition into conjuncts, as suggested above, would also avoid the product here, and stays useful after CALCITE-7869: when the conjuncts relate different unions, as in a chain of joins over several unions, the count becomes a sum instead of a product. The two changes complement each other: CALCITE-7869 removes the combinations that cannot occur, including within a single expression, and splitting keeps the valid ones small.


> RelMdExpressionLineage#createAllPossibleExpressions  will create many RexCall lead to OOM 
> ------------------------------------------------------------------------------------------
>
>                 Key: CALCITE-5365
>                 URL: https://issues.apache.org/jira/browse/CALCITE-5365
>             Project: Calcite
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 1.30.0
>            Reporter: hqbhoho
>            Priority: Major
>
> Execute query like:
> {code:java}
> select
> *
> from
> (
> select *
> from
> (
> select * from A1
> )
> union all
> (
> select * from A2
> )
> union all
> (
> select * from A3
> )
> ) t1
> left join
> (
> select *
> from
> (
> select * from B1
> )
> union all
> (
> select * from B2
> )
> union all
> (
> select * from B3
> )
> )t2
> on t1.a1 = t2.b1
> and t1.a2 = t2.b2
> and t1.a3 = t2.b3
> and t1.a4 = t2.b4
> and t1.a5 = t2.b5
> and t1.a6 = t2.b6
> and t1.a7 = t2.b7
> left join C1 name
> on t2.b8=name.c1 {code}
> I will get all Predicates via RelMetadataQuery.getAllPredicates()，but will occur oom. 
> RelMdExpressionLineage#createAllPossibleExpressions will retrun 3^14 RexCall when t1 join t2 with 7 condition? (t1.a1 will have 3 replacement)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)