[jira] [Updated] (SPARK-41535) InterpretedUnsafeProjection and InterpretedMutableProjection can corrupt unsafe buffer when used with calendar interval data

["Bruce Robbins (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-41535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bruce Robbins updated SPARK-41535:
----------------------------------
    Affects Version/s: 3.2.3

> InterpretedUnsafeProjection and InterpretedMutableProjection can corrupt unsafe buffer when used with calendar interval data
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-41535
>                 URL: https://issues.apache.org/jira/browse/SPARK-41535
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.3.1, 3.2.3, 3.4.0
>            Reporter: Bruce Robbins
>            Priority: Major
>
> This returns the wrong answer:
> {noformat}
> set spark.sql.codegen.wholeStage=false;
> set spark.sql.codegen.factoryMode=NO_CODEGEN;
> select first(col1), last(col2) from values
> (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0))
> as data(col1, col2);
> +---------------+---------------+
> |first(col1)    |last(col2)     |
> +---------------+---------------+
> |16 years 2 days|16 years 2 days|
> +---------------+---------------+
> {noformat}
> In the above case, {{TungstenAggregationIterator}} uses {{InterpretedUnsafeProjection}} to create the aggregation buffer and then initializes all the fields to null. {{InterpretedUnsafeProjection}} incorrectly calls {{UnsafeRowWriter#setNullAt}}, rather than {{unsafeRowWriter#write}}, for the two calendar interval fields. As a result, the writer never allocates memory from the variable length region for the two decimals, and the pointers in the fixed region get left as zero. Later, when {{InterpretedMutableProjection}} attempts to update the first field, {{UnsafeRow#setInterval}} picks up the zero pointer and stores interval data on top of the null-tracking bit set. The call to UnsafeRow#setInterval for the second field also stomps the null-tracking bit set. Later updates to the null-tracking bit set (e.g., calls to setNotNullAt) further corrupt the interval data, turning {{interval 7 years 2 days}} into {{interval 16 years 2 days}}.
> Even if you fix the above bug to {{InterpretedUnsafeProjection}} so that the buffer is created correctly, {{InterpretedMutableProjection}} has a similar bug to SPARK-41395, except this time for calendar interval data:
> {noformat}
> set spark.sql.codegen.wholeStage=false;
> set spark.sql.codegen.factoryMode=NO_CODEGEN;
> select first(col1), last(col2), max(col3) from values
> (null, null, 1),
> (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
> as data(col1, col2, col3);
> +---------------+---------------+---------+
> |first(col1)    |last(col2)     |max(col3)|
> +---------------+---------------+---------+
> |16 years 2 days|16 years 2 days|3        |
> +---------------+---------------+---------+
> {noformat}
> These two bugs could get exercised during codegen fallback. Take for example this case where I forced codegen to fail for the Greatest expression:
> {noformat}
> spark-sql> select first(col1), last(col2), max(col3) from values
> (null, null, 1),
> (make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
> as data(col1, col2, col3);
> 22/12/15 13:06:23 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 70, Column 1: ';' expected instead of 'if'
> ...
> 22/12/15 13:06:24 WARN MutableProjection: Expr codegen error and falling back to interpreter mode
> java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 78, Column 1: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 78, Column 1: ';' expected instead of 'boolean'
> ...
> 16 years 2 days	16 years 2 days	3
> Time taken: 5.852 seconds, Fetched 1 row(s)
> spark-sql> 
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org