[jira] [Updated] (SPARK-59097) UnsafeProjection codegen throws NullPointerException on a null in a nested non-nullable field, where the interpreted path writes null

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59097:
-----------------------------------
    Labels: pull-request-available  (was: )

> UnsafeProjection codegen throws NullPointerException on a null in a nested non-nullable field, where the interpreted path writes null
> -------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59097
>                 URL: https://issues.apache.org/jira/browse/SPARK-59097
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.2, 4.2.0, 5.0.0
>            Reporter: Joel Robin
>            Priority: Major
>              Labels: pull-request-available
>
> A struct, array or map value that is declared non-nullable but is actually null at runtime gets handled two different ways depending on which UnsafeProjection you end up with. The interpreted one writes null and carries on. The generated one dereferences the null and throws. Both are implementations of the same thing, so which one you get should not decide whether the query survives.
> It surfaces as a task failure like this:
> {code}
> java.lang.NullPointerException: Cannot invoke
>   "org.apache.spark.sql.catalyst.util.ArrayData.numElements()" because "<local10>" is null
>   at ...GeneratedClass$SpecificUnsafeProjection.writeFields_1_3$(Unknown Source)
>   at ...GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
>   at ...GeneratedClass$GeneratedIteratorForCodegenStage31.processNext(Unknown Source)
>   at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:154)
> {code}
> h3. Why codegen throws
> writeExpressionsToBuffer drops the null check whenever the declared nullability says the value cannot be null (GenerateUnsafeProjection.scala:156):
> {code:scala}
> if (!nullable || input.isNull == FalseLiteral) {
>   // The value is statically known to be non-null, so skip the null check and the
>   // (dead) setNull branch and just write the value.
> {code}
> Array elements get the same treatment when containsNull is false (:248-249). writeArrayToBuffer then does the dereference (:257):
> {code:java}
> final ArrayData tmpInput = <input>;
> if (tmpInput instanceof UnsafeArrayData) { ... }
> else { final int numElements = tmpInput.numElements();   // throws here
> {code}
> tmpInput is the <localN> named in the NPE message.
> InterpretedUnsafeProjection.generateFieldWriter does the opposite, and the comment is explicit about it (InterpretedUnsafeProjection.scala:251):
> {code:scala}
> // Always wrap the writer with a null safe version.
> {code}
> It does take nullable as a parameter, but only to work out the child writer's nullability. It
> never consults it when deciding whether to null-check. So the interpreted path tolerates an
> inaccurate declaration and codegen does not.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org