You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Avery Qi (Jira)" <ji...@apache.org> on 2024/08/26 18:22:00 UTC
[jira] [Created] (SPARK-49396) PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule
Avery Qi created SPARK-49396:
--------------------------------
Summary: PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule
Key: SPARK-49396
URL: https://issues.apache.org/jira/browse/SPARK-49396
Project: Spark
Issue Type: Task
Components: Optimizer
Affects Versions: 4.0.0
Reporter: Avery Qi
{{}}
{{SimplifyConditionals}} has a simplification code where if any of the (not first) branches in {{CaseWhen}} is {{TrueLiteral}} we remove all the remaining branches including the {{{}elseValue{}}}.
{{ case CaseWhen(branches, _) if branches.exists(_._1 == TrueLiteral) => // a branch with a true condition eliminates all following branches,
// these branches can be pruned away
val (h, t) = branches.span(_._1 != TrueLiteral)
CaseWhen( h :+ t.head, None)}}
Now, the nullability check of {{CaseWhen}} checks that
(1) either of the branches including {{elseValue}} is {{nullable}} or
(2) {{elseValue}} is {{{}None{}}}.
The above simplification makes the {{elseValue}} as {{{}None{}}}. Combined with this nullability check makes the {{CaseWhen}} switch from {{non-nullable}} to {{{}nullable{}}}.
This ticket aims to fix this issue and remain the nullability of the expression by replacing the elsevalue to the value of the TrueLiteral expression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org