[jira] [Commented] (SPARK-39839) Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569847#comment-17569847 ] 

Apache Spark commented on SPARK-39839:
--------------------------------------

User 'rednaxelafx' has created a pull request for this issue:
https://github.com/apache/spark/pull/37252

> Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-39839
>                 URL: https://issues.apache.org/jira/browse/SPARK-39839
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.1.0, 3.2.0, 3.3.0
>            Reporter: Kris Mok
>            Priority: Major
>
> The {{UnsafeRow}} structural integrity check in {{UnsafeRowUtils.validateStructuralIntegrity}} is added in Spark 3.1.0. It’s supposed to validate that a given {{UnsafeRow}} conforms to the format that the {{UnsafeRowWriter}} would have produced.
> Currently the check expects all fields that are marked as null should also have its field (i.e. the fixed-length part) set to all zeros. It needs to be updated to handle a special case for variable-length {{{}Decimal{}}}s, where the {{UnsafeRowWriter}} may mark a field as null but also leave the fixed-length part of the field as {{OffsetAndSize(offset=current_offset, size=0)}}. This may happen when the {{Decimal}} being written is either a real {{null}} or has overflowed the specified precision.
> Logic in {{UnsafeRowWriter}}:
> in general:
> {code:scala}
>   public void setNullAt(int ordinal) {
>     BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
>     write(ordinal, 0L);                                      // also zero out the fixed-length field
>   } {code}
> special case for {{DecimalType}}:
> {code:scala}
>       // Make sure Decimal object has the same scale as DecimalType.
>       // Note that we may pass in null Decimal object to set null for it.
>       if (input == null || !input.changePrecision(precision, scale)) {
>         BitSetMethods.set(getBuffer(), startingOffset, ordinal); // set null bit
>         // keep the offset for future update
>         setOffsetAndSize(ordinal, 0);                            // doesn't zero out the fixed-length field
>       } {code}
> The special case is introduced to allow all {{DecimalType}}s (including both fixed-length and variable-length ones) to be mutable – thus need to leave space for the variable-length field even if it’s currently null.
> Note that this special case in {{UnsafeRowWriter}} has been there since Spark 1.6.0, where as the integrity check was added in Spark 3.1.0. The check was originally added for Structured Streaming’s checkpoint evolution validation, so that a newer version of Spark can check whether or not an older checkpoint file for Structured Streaming queries can be supported, and/or if the contents of the checkpoint file is corrupted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org