[jira] [Created] (SPARK-60418) Subexpression elimination evaluates a short-circuited IN list element

["sepuri sai krishna (Jira)" <ji...@apache.org>]

sepuri sai krishna created SPARK-60418:
------------------------------------------

             Summary: Subexpression elimination evaluates a short-circuited IN list element
                 Key: SPARK-60418
                 URL: https://issues.apache.org/jira/browse/SPARK-60418
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: sepuri sai krishna


{{In}} stops at the first list element that matches, in both {{eval}} and
generated code, but a later element is still evaluated ahead of the comparison
when it is a common subexpression. A query that should return rows then fails.

ANSI mode, default configuration:

{code:sql}
CREATE OR REPLACE TEMP VIEW t AS
  SELECT cast(id as int) AS b, cast(2147483647 - id as int) AS y FROM range(1);

SELECT b IN (b, y + 1), b IN (b, y + 1, 7) FROM t;
-- b matches the first element of both lists, so y + 1 is never needed
-- expected: true, true
-- actual:   [ARITHMETIC_OVERFLOW] overflow
{code}

Either {{IN}} on its own returns {{true}}, and so does the query with
{{spark.sql.subexpressionElimination.enabled}} set to {{false}}, so the error
comes from the elimination rather than from the query. It fails with whole-stage
codegen both on and off. A list of literals is not affected.

{{spark.sql.subexpressionElimination.skipForShortcutExpr}}, which exists to stop
elimination from evaluating short-circuited operands, does not help here: with it
set to {{true}} the same shape over {{AND}} stops failing, while the {{IN}} query
above still fails. {{In}} short-circuits the same way, and is already listed
alongside {{And}} and {{Or}} as conditionally evaluated in
{{OptimizeCsvJsonExprs}}.

Verified on master at 8d5b30a06f6.

---

## Why only one

The try_* candidate does not hold up. {{try_subtract(cast(s as date), cast(s as
date))}} does raise with elimination on and return NULL with it off, but NULL is
not what try_* promises:

- the numeric path, {{try_subtract(cast(s as int), 1)}}, raises the operand's
  CAST_INVALID_INPUT in every mode, so try_* does not swallow operand errors by
  design;
- what the docs promise is NULL on the operation's own overflow, and
  {{try_add(date '2020-01-01', make_ym_interval(178956970, 0))}} returns NULL in
  every mode, elimination on or off;
- the non-numeric path swallows operand errors only incidentally, because
  {{TryEval}} wraps the whole subtree, and the code carries a
  "TODO: support TRY eval mode on datetime arithmetic expressions" there.

So with elimination on, the non-numeric path becomes consistent with the numeric
one. Filing it as "try_* raises instead of returning NULL" would be arguing for
the anomalous behaviour.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org