[jira] [Updated] (SPARK-60021) Label ExpandExec's outputs held in mutable state as global fields in codegen

["Max Gekk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Max Gekk updated SPARK-60021:
-----------------------------
    Affects Version/s: 4.2.0
                       4.1.3
                       4.3.0
                           (was: 5.0.0)

> Label ExpandExec's outputs held in mutable state as global fields in codegen
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-60021
>                 URL: https://issues.apache.org/jira/browse/SPARK-60021
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0, 4.3.0, 4.1.3
>            Reporter: Max Gekk
>            Priority: Major
>
> Since SPARK-35329, ExpandExec keeps its outputs in mutable state fields (addMutableState), which may be compacted into an array slot such as expand_mutableStateArray_0[k] for non-primitive types. But it labels them as local variables, JavaCode.isNullVariable(isNull) and JavaCode.variable(value, ...) in ExpandExec.doConsume, while every other producer that keeps its outputs in fields labels them isNullGlobal/global.
> As a result, CodegenContext.getLocalInputVariableValues takes such a slot as a local input, and a method split out of the stage (e.g. a subexpression elimination method) declares it as a parameter. No Java parameter can be named "array[i]", so the generated code fails to compile and the stage falls back to the non-codegen path (or fails when the fallback is off). For example, with Janino:
>   ')' expected instead of '['
> Example query:
>   CREATE TEMP VIEW t AS SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS STRING)) AS b FROM range(10);
>   SELECT c,
>     concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
>     concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z') AS y
>   FROM t UNPIVOT (v FOR c IN (a, b));
> Released 4.1.3, 4.2.0 and 4.3.0 fail the same way.
> Proposed fix: label ExpandExec's outputs as JavaCode.isNullGlobal / JavaCode.global, so methods split out of the stage read them as fields. This came up in the review of https://github.com/apache/spark/pull/59225 (SPARK-33301), which currently works around it with a compacted-slot rule; once this is fixed, that PR can drop the workaround.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org