You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bruce Robbins (Jira)" <ji...@apache.org> on 2022/06/16 23:30:00 UTC
[jira] [Created] (SPARK-39496) Inline eval path cannot handle null structs
Bruce Robbins created SPARK-39496:
-------------------------------------
Summary: Inline eval path cannot handle null structs
Key: SPARK-39496
URL: https://issues.apache.org/jira/browse/SPARK-39496
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1, 3.1.3, 3.3.0, 3.4.0
Reporter: Bruce Robbins
This issue is somewhat similar to SPARK-39061, but for the eval path rather than the codegen path.
Example:
{noformat}
set spark.sql.codegen.wholeStage=false;
select inline(array(named_struct('a', 1, 'b', 2), null));
{noformat}
This results in a NullPointerException:
{noformat}
22/06/16 15:10:06 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
{noformat}
The next example doesn't require setting {{spark.sql.codegen.wholeStage}} to {{{}false{}}}:
{noformat}
val dfWide = (Seq((1))
.toDF("col0")
.selectExpr(Seq.tabulate(99)(x => s"$x as col${x + 1}"): _*))
val df = (dfWide
.selectExpr("*", "array(named_struct('a', 1, 'b', 2), null) as struct_array"))
df.selectExpr("*", "inline(struct_array)").collect
{noformat}
The result is similar:
{noformat}
22/06/16 15:18:55 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1]
java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(JoinedRow.scala:80)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_8$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.execution.GenerateExec.$anonfun$doExecute$11(GenerateExec.scala:122)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org