You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ahshahid (via GitHub)" <gi...@apache.org> on 2026/01/16 19:53:34 UTC
[PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
ahshahid opened a new pull request, #53838:
URL: https://github.com/apache/spark/pull/53838
…romConstraints rule are run as part of the operatorOptimizationRuleSet, so that generated filter on Left leg of Join, in some specific situation, is not missed
### What changes were proposed in this pull request?
The change is to merge the batch
```
Batch("Infer Filters", Once,
InferFiltersFromGenerate,
InferFiltersFromConstraints)
```
into the batch of rules defined by
"operatorOptimizationRuleSet".
### Why are the changes needed?
The way current set of optimizer rules work is that
```
// **step1**
Batch("Operator Optimization before Inferring Filters", fixedPoint,
operatorOptimizationRuleSet: _*),
// **step2**
Batch("Infer Filters", Once,
InferFiltersFromGenerate,
InferFiltersFromConstraints),
// **step3**
Batch("Operator Optimization after Inferring Filters", fixedPoint,
operatorOptimizationRuleSet: _*)
```
In the batch of rules "operatorOptimizationRuleSet", the conversion of Joins like "Left Outer" to Inner happens.{}
After that "InferFiltersFromConstraints" is called which is able to create new constraints like IsNotNull, to be pushed on either side of the Inner Join tables.{}
Notice that "operatorOptimizationRuleSet" is called twice, before and after inferring filters.{}
It so happens that in TPCDS Q5, atleast, the conversion of LeftOuter to Inner for one of the Join cases, happens in step3.
But since, there is no further call of InferFiltersFromConstraints, the IsNotNull constraints generation is missed, for the Left Leg of the Join.
Please note that the numerous plan changes are cosmetic ( caused by reordering of text), except in case of q5, where two new not null constraints are created for the Left Leg of the Join.
I will try to fix the cosmetic change issue though might be difficult ..
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Passing of existing tests..
will be adding a dedicated bug test.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3801655387
The combined PR for SPARK-55185 and SPARK-55072 is
[PR-SPARK-55072_SPARK-55185](https://github.com/apache/spark/pull/53983)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3761575443
## JIRA Issue Information
=== Improvement SPARK-55072 ===
Summary: Inferring new Constraint misses IsNotNull on Left Leg, when an Outer Join gets converted into inner Join
Assignee: None
Status: Open
Affected: ["4.2.0","4.1.1"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-4414094218
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #53838: [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer)
URL: https://github.com/apache/spark/pull/53838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3801702578
The clean pass for this PR is contingent on the PR for SPARK-55185 ( [PR_SPARK_55185](https://github.com/apache/spark/pull/53968)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on code in PR #53838:
URL: https://github.com/apache/spark/pull/53838#discussion_r2699832388
##########
sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/simplified.txt:
##########
@@ -110,10 +110,11 @@ TakeOrderedAndProject [channel,id,sales,returns,profit]
InputAdapter
BroadcastExchange #8
WholeStageCodegen (14)
- ColumnarToRow
- InputAdapter
- Scan parquet spark_catalog.default.web_returns [wr_item_sk,wr_order_number,wr_return_amt,wr_net_loss,wr_returned_date_sk]
- ReusedSubquery [d_date_sk] #1
+ Filter [wr_item_sk,wr_order_number]
Review Comment:
This is the IsNotNull filter which is missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3761831365
There are multiple failures.. especially like ThriftServerQueryTestSuite, which to me indicates other hidden bugs .. looking into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]
Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3816247694
The combined PR [SPARK-55072 + SPARK-55185 + SPARK-55241](https://github.com/apache/spark/pull/53983) is passing, so this PR is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org