You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Avery Qi (Jira)" <ji...@apache.org> on 2024/08/26 18:22:00 UTC

[jira] [Created] (SPARK-49396) PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule

Avery Qi created SPARK-49396:
--------------------------------

             Summary: PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule
                 Key: SPARK-49396
                 URL: https://issues.apache.org/jira/browse/SPARK-49396
             Project: Spark
          Issue Type: Task
          Components: Optimizer
    Affects Versions: 4.0.0
            Reporter: Avery Qi


 
{{}}
{{SimplifyConditionals}} has a simplification code where if any of the (not first) branches  in {{CaseWhen}} is {{TrueLiteral}} we remove all the remaining branches including the {{{}elseValue{}}}.
 {{      case CaseWhen(branches, _) if branches.exists(_._1 == TrueLiteral) =>        // a branch with a true condition eliminates all following branches,
        // these branches can be pruned away
        val (h, t) = branches.span(_._1 != TrueLiteral)
        CaseWhen( h :+ t.head, None)}}
Now, the nullability check of {{CaseWhen}} checks that 
(1) either of the branches including {{elseValue}} is {{nullable}} or 
(2) {{elseValue}} is {{{}None{}}}.

The above simplification makes the {{elseValue}} as {{{}None{}}}. Combined with this nullability check makes the {{CaseWhen}} switch from {{non-nullable}} to {{{}nullable{}}}.

This ticket aims to fix this issue and remain the nullability of the expression by replacing the elsevalue to the value of the TrueLiteral expression.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org