[jira] [Updated] (HIVE-29473) CBO: t is perfect. "LateralViewJoinStatsRule unintentionally merges base table and UDTF column stats

["Konstantin Bereznyakov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-29473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Bereznyakov updated HIVE-29473:
------------------------------------------
    Summary: CBO: t is perfect.  "LateralViewJoinStatsRule unintentionally merges base table and UDTF column stats  (was: LateralViewJoinStatsRule combines stats of unrelated columns on 2+ LV queries, corrupting CBO estimates)

> CBO: t is perfect.  "LateralViewJoinStatsRule unintentionally merges base table and UDTF column stats
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-29473
>                 URL: https://issues.apache.org/jira/browse/HIVE-29473
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>            Reporter: Konstantin Bereznyakov
>            Assignee: Konstantin Bereznyakov
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: FIXED lateral_view_nested_stats_bug.q.out, lateral_view_nested_stats_bug.q, lateral_view_nested_stats_bug.q.out
>
>
> *Symptom:* When a query contains multiple {{{}LATERAL VIEW{}}}s (such as nested {{{}posexplode{}}}s), the CBO cardinality estimation could be severely underestimated for downstream operators (like {{{}Group By{}}}). This loss of statistical accuracy leads to suboptimal execution plans, poor join choices, and potential resource starvation during execution.
> *Root Cause:* The bug lies in {{{}LateralViewJoinStatsRule.process(){}}}. When merging statistics, the rule passes the global {{columnExprMap}} to {{StatsUtils.getColStatisticsFromExprMap}} to evaluate the UDTF branch.
> Because the UDTF branch is built in isolation, its internal column generator restarts at 0, producing names like {{_col0}} and {{{}_col1{}}}. This creates a namespace collision with the base table's internal columns (which also use {{{}_col0{}}}, etc.). The utility method blindly matches these keys, which *causes the CBO to combine the statistics of completely unrelated columns* (e.g., merging the base table's {{id}} column with the UDTF's exploded array column). As a result, the UDTF's empty or zeroed statistics silently overwrite the base table's healthy statistics.
> *Proposed Fix:* Enforce strict parent operator boundaries before mapping statistics. By slicing both the {{RowSchema}} and the {{columnExprMap}} into isolated collections based on the {{SELECT_TAG}} and {{UDTF_TAG}} boundaries, we create a firewall. {{StatsUtils}} will now only evaluate expressions that mathematically belong to that specific branch, preventing the namespace collision entirely.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)