You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/08/18 00:42:43 UTC

[PR] [SPARK-51356][SQL] Emit IsNotNull on a complex child before the FilterExec predicate that uses it [spark]

jiwen624 opened a new pull request, #58058:
URL: https://github.com/apache/spark/pull/58058

   ### What changes were proposed in this pull request?
   Currently `FilterExec` matches an `otherPred` to its guarding `IsNotNull` by attribute reference, so an `IsNotNull` over a complex child (`IsNotNull(b.c)`) never matches and is deferred until after the predicates that use it. This fix emits it just before an otherPred containing that child, in both `generatePredicateCode` and the CSE branch of `doConsume` (which must stay in lockstep).
   
   ### Why are the changes needed?
   The guard runs after the predicate it guards:
   
   ```scala
   case class C(d: Int); case class B(c: C = null); case class A(b: B = null)
   val ds = spark.createDataset(Seq(A(null), A(B(null)), A(B(C(0))), A(B(C(1))))).map(identity)
     .where($"b".isNotNull).select($"b".as[B])
   val isDZero = udf((c: C) => c.d == 0)
   ds.filter($"c".isNotNull).filter(not(isDZero($"c"))).show()
   ```
   
   ScalaUDF passes null to its deserializer instead of short-circuiting, so this throws EXPRESSION_DECODING_FAILED under whole-stage codegen while interpreted mode returns the correct row. Only UDFs taking the nested struct are affected.
   
   ### Does this PR introduce any user-facing change?
   No API change. The failed query above now succeeds.
   
   ### How was this patch tested?
   Unit test cases added.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   Yes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51356][SQL] Emit IsNotNull on a complex child before the FilterExec predicate that uses it [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58058:
URL: https://github.com/apache/spark/pull/58058#issuecomment-5352160678

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/eaf9d3a5da52d9d3180c84a744219baa3074c2dc
   - merged into branch-4.x https://github.com/apache/spark/commit/f4ad8a5d0d7da63b4039d3445df2eeb5527a7ede
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51356][SQL] Emit IsNotNull on a complex child before the FilterExec predicate that uses it [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #58058: [SPARK-51356][SQL] Emit IsNotNull on a complex child before the FilterExec predicate that uses it
URL: https://github.com/apache/spark/pull/58058


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-51356][SQL] Emit IsNotNull on a complex child before the FilterExec predicate that uses it [spark]

Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #58058:
URL: https://github.com/apache/spark/pull/58058#issuecomment-5349927032

   Hi @hvanhovell @cloud-fan @uros-b could you take a look at this patch when you get a chance? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org