[jira] [Updated] (SPARK-58431) SimplifyExtractValueOps folds an out-of-bounds array index to NULL under ANSI, suppressing INVALID_ARRAY_INDEX and corrupting the result's nullability

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-58431:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> SimplifyExtractValueOps folds an out-of-bounds array index to NULL under ANSI, suppressing INVALID_ARRAY_INDEX and corrupting the result's nullability
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58431
>                 URL: https://issues.apache.org/jira/browse/SPARK-58431
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Josh Rosen
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> This is a report of a correctness bug reachable under default configuration on Spark 4.x (ANSI mode is now enabled by default). A query that must raise {{INVALID_ARRAY_INDEX}} instead returns a fabricated value of {{{}0{}}}, silently.
>  
> {{SimplifyExtractValueOps}} rewrites {{array(...)[i]}} when the index is a literal ({{{}ComplexTypes.scala{}}}):
>  
> {code:java}
> case ga @ GetArrayItem(CreateArray(elems, _), IntegerLiteral(idx), _) =>
>   if (idx >= 0 && idx < elems.size) {
>     elems(idx)
>   } else {
>     // out of bounds, mimic the runtime behavior and return null
>     Literal(null, ga.dataType)
>   } {code}
> The comment states the intent: mimic the runtime behavior. That was accurate when the rule was written (SPARK-18601, 2.2.0), but SPARK-33386 (3.1.1) changed the runtime behavior: under ANSI, an out-of-bounds array access _raises_ rather than returning null. The rule was never updated, and it does not consult {{GetArrayItem.failOnError}} (which defaults to {{{}SQLConf.get.ansiEnabled{}}}).
>  
> This causes two problems:
>  * The error is suppressed.
>  * Under ANSI {{GetArrayItem}} declares itself *non-nullable* (it throws instead of returning null), so substituting a null literal places a NULL in a slot the schema marks non-nullable. Depending on the downstream consumer, this can return either NULL (incorrect under ANSI) or as the type's zero value (also incorrect).
>  
> *Repro* (default configuration on 4.x; add {{SET spark.sql.ansi.enabled=true}} on older versions):
>  
> {code:java}
> SET spark.sql.ansi.enabled=true;
> CREATE TABLE arr_src(c INT) USING parquet;
> INSERT INTO arr_src VALUES (1);
> SELECT array(c, 2, 3)[5] AS v FROM arr_src;
> -- prints: NULL       (expected under ANSI: INVALID_ARRAY_INDEX error)
> SELECT count(*) FROM arr_src WHERE array(c, 2, 3)[5] IS NULL;
> -- prints: 0          (the value that just printed NULL matches no IS NULL filter)
> SELECT array(c, 2, 3)[5] IS NULL AS n FROM arr_src;
> -- prints: false
> SELECT sum(array(c,2,3)[5]) AS s, count(array(c,2,3)[5]) AS n FROM arr_src;
> -- prints: NULL, 1    (count treats it as one NON-null value; sum of it is null) {code}
> No consistent value can produce those four outputs together. {*}Which symptom the corrupted slot surfaces as is read-path-dependent{*}: the spark-sql CLI's row rendering honors the physical null bit and prints {{{}NULL{}}}, while {{Dataset.collect()}} (Scala/Java/Python APIs) trusts the declared {{{}nullable = false{}}}, elides the null check, and materializes a fabricated {{0.}}
>  
> Affected versions: the fold dates from 2.2.0, but it only became incorrect when SPARK-33386 (3.1.1) made ANSI out-of-bounds access raise. So 3.1.1+ with ANSI enabled, and {*}4.x by default{*}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org