[jira] [Assigned] (HIVE-30125) Log missing column stats at WARN instead of ERROR in RelOptHiveTable

["Konstantin Bereznyakov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Bereznyakov reassigned HIVE-30125:
---------------------------------------------

    Assignee: Konstantin Bereznyakov

> Log missing column stats at WARN instead of ERROR in RelOptHiveTable
> --------------------------------------------------------------------
>
>                 Key: HIVE-30125
>                 URL: https://issues.apache.org/jira/browse/HIVE-30125
>             Project: Hive
>          Issue Type: Improvement
>          Components: CBO
>            Reporter: Konstantin Bereznyakov
>            Assignee: Konstantin Bereznyakov
>            Priority: Minor
>
> When CBO needs column statistics that are missing, {{RelOptHiveTable.updateColStats}} logs {{{}No Stats for <db>@<table>, Columns: <cols>{}}}. Whether this is a WARN or an ERROR depends on the {{allowMissingStats}} flag:
> {code:java}
> if (allowMissingStats) {
>   LOG.warn(logMsg);
>   ...
> } else {
>   LOG.error(logMsg);
>   throw new RuntimeException(logMsg);
> }
> {code}
> {{allowMissingStats}} is false when {{{}hive.stats.estimate=false{}}}, and it is always false in {{HiveReduceExpressionsWithStatsRule}} (enabled by {{{}hive.optimize.filter.stats.reduction{}}}). In these cases, the same missing stats that are normally a WARN are logged as an ERROR.
> The thrown exception is the real signal, and the callers already log the result at the right level:
>  * {{CalcitePlanner}} (join reordering), {{HiveAggregateJoinTransposeRule}} and {{HiveFilterSortPredicates}} catch the exception, log a WARN ({{{}Missing column stats (see previous messages), skipping ...{}}}) and continue planning.
>  * If the exception is not caught, {{CalcitePlanner}} logs {{CBO failed, skipping CBO.}} at ERROR with the exception, which includes the same table and column names. Depending on {{{}hive.cbo.fallback.strategy{}}}, the query then fails or is retried without CBO.
> So this ERROR never tells us anything new. If planning recovers, a normal WARN-level situation shows up as an ERROR. If planning fails, it repeats the ERROR that {{CalcitePlanner}} already logs. Tools that alert on ERROR logs then report normal, recovered planning as a failure. We see a large number of these in a deployment that runs with {{{}hive.stats.estimate=false{}}}.
> History: the ERROR and the throw come from the original CBO work (HIVE-7395, HIVE-7918), when missing stats always stopped CBO. HIVE-10315 split the code into a WARN branch and a strict branch. Later, HIVE-12017 and HIVE-22163 added callers that catch the exception and continue planning. The log level in the strict branch was not changed when the exception became recoverable.
> *Proposed change:*
>  * {{{}RelOptHiveTable.updateColStats{}}}: log the message at WARN in both branches. The exception and the control flow stay the same.
>  * {{{}CalcitePlanner{}}}: change {{(see previous errors)}} to {{(see previous messages)}} in the missing-stats CBO failure message, to match the other three call sites.
> There is no behavior change. The same exception is thrown and handled in the same way, and the message with the table and column names is still logged at WARN.
> Checked against master at 744271738e.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)