You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/06/17 19:45:00 UTC
[jira] [Commented] (SPARK-39496) Inline eval path cannot handle null structs
[ https://issues.apache.org/jira/browse/SPARK-39496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555746#comment-17555746 ]
Apache Spark commented on SPARK-39496:
--------------------------------------
User 'bersprockets' has created a pull request for this issue:
https://github.com/apache/spark/pull/36903
> Inline eval path cannot handle null structs
> -------------------------------------------
>
> Key: SPARK-39496
> URL: https://issues.apache.org/jira/browse/SPARK-39496
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.3, 3.2.1, 3.3.0, 3.4.0
> Reporter: Bruce Robbins
> Priority: Major
>
> This issue is somewhat similar to SPARK-39061, but for the eval path rather than the codegen path.
> Example:
> {noformat}
> set spark.sql.codegen.wholeStage=false;
> select inline(array(named_struct('a', 1, 'b', 2), null));
> {noformat}
> This results in a NullPointerException:
> {noformat}
> 22/06/16 15:10:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
> java.lang.NullPointerException
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
> at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
> {noformat}
> The next example doesn't require setting {{spark.sql.codegen.wholeStage}} to {{{}false{}}}:
> {noformat}
> val dfWide = (Seq((1))
> .toDF("col0")
> .selectExpr(Seq.tabulate(99)(x => s"$x as col${x + 1}"): _*))
> val df = (dfWide
> .selectExpr("*", "array(named_struct('a', 1, 'b', 2), null) as struct_array"))
> df.selectExpr("*", "inline(struct_array)").collect
> {noformat}
> The result is similar:
> {noformat}
> 22/06/16 15:18:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
> java.lang.NullPointerException
> at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:80)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_8$(Unknown Source)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
> at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org