You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2024/09/19 15:30:00 UTC

[jira] [Updated] (IMPALA-13302) Some ExprRewriteRule results are not analyzed, leading to unmaterialized slots from reAnalyze

     [ https://issues.apache.org/jira/browse/IMPALA-13302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Smith updated IMPALA-13302:
-----------------------------------
    Description: 
IMPALA-12164 skipped registering conjuncts that the analyzer expects to remove because an earlier conjunct evaluates to constant False. However some ExprRewriteRules don't analyze the predicates they produce, which can lead to those conjuncts not actually being removed until a reAnalyze phase.

reAnalyze uses a new Analyzer (with new GlobalState); it restarts counting Expr IDs from 0. That can lead to re-using the same Expr ID and marking it as assigned. Then when a new Expr gets the same ID, it will skip materializing slots, which can cause problems later (like if that Expr is part of a hash join).

Some example queries:
1. still a problem
{code}
WITH v AS (SELECT 1 FROM functional.alltypestiny t1
  JOIN functional.alltypestiny t2 ON t1.id = t2.id)
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;
{code}
2. already fixed via IMPALA-13203
{code}
WITH v as (SELECT 1 FROM functional.alltypes t1
  JOIN functional.alltypes t2 ON t1.id = t2.id)
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND t1.id = 1 AND t1.id = 1 AND false
UNION ALL
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND false
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;
{code}


  was:
IMPALA-12164 skipped registering conjuncts that the analyzer expects to remove because an earlier conjunct evaluates to constant False. However some ExprRewriteRules don't analyze the predicates they produce, which can lead to those conjuncts not actually being removed until a reAnalyze phase.

reAnalyze uses a new Analyzer (with new GlobalState); it restarts counting Expr IDs from 0. That can lead to re-using the same Expr ID and marking it as assigned. Then when a new Expr gets the same ID, it will skip materializing slots, which can cause problems later (like if that Expr is part of a hash join).

Some example queries:
{code}
WITH v AS (SELECT 1 FROM functional.alltypestiny t1
  JOIN functional.alltypestiny t2 ON t1.id = t2.id)
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL
SELECT 1
FROM functional.alltypestiny t1
WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
  AND t1.id = 1 AND t1.id = 1
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;
{code}
(already fixed via IMPALA-13203) and
{code}
WITH v as (SELECT 1 FROM functional.alltypes t1
  JOIN functional.alltypes t2 ON t1.id = t2.id)
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND t1.id = 1 AND t1.id = 1 AND false
UNION ALL
SELECT 1 FROM functional.alltypes t1
  WHERE t1.id = 1 AND false
UNION ALL SELECT 1 FROM v
UNION ALL SELECT 1 FROM v;
{code}



> Some ExprRewriteRule results are not analyzed, leading to unmaterialized slots from reAnalyze
> ---------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-13302
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13302
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>    Affects Versions: Impala 4.3.0
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Critical
>
> IMPALA-12164 skipped registering conjuncts that the analyzer expects to remove because an earlier conjunct evaluates to constant False. However some ExprRewriteRules don't analyze the predicates they produce, which can lead to those conjuncts not actually being removed until a reAnalyze phase.
> reAnalyze uses a new Analyzer (with new GlobalState); it restarts counting Expr IDs from 0. That can lead to re-using the same Expr ID and marking it as assigned. Then when a new Expr gets the same ID, it will skip materializing slots, which can cause problems later (like if that Expr is part of a hash join).
> Some example queries:
> 1. still a problem
> {code}
> WITH v AS (SELECT 1 FROM functional.alltypestiny t1
>   JOIN functional.alltypestiny t2 ON t1.id = t2.id)
> SELECT 1
> FROM functional.alltypestiny t1
> WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
>   AND t1.id = 1 AND t1.id = 1
> UNION ALL
> SELECT 1
> FROM functional.alltypestiny t1
> WHERE ((t1.id = 1 and false) or (t1.id = 1 and false))
>   AND t1.id = 1 AND t1.id = 1
> UNION ALL SELECT 1 FROM v
> UNION ALL SELECT 1 FROM v;
> {code}
> 2. already fixed via IMPALA-13203
> {code}
> WITH v as (SELECT 1 FROM functional.alltypes t1
>   JOIN functional.alltypes t2 ON t1.id = t2.id)
> SELECT 1 FROM functional.alltypes t1
>   WHERE t1.id = 1 AND t1.id = 1 AND t1.id = 1 AND false
> UNION ALL
> SELECT 1 FROM functional.alltypes t1
>   WHERE t1.id = 1 AND false
> UNION ALL SELECT 1 FROM v
> UNION ALL SELECT 1 FROM v;
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org