You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ahshahid (via GitHub)" <gi...@apache.org> on 2026/01/16 19:53:34 UTC

[PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

ahshahid opened a new pull request, #53838:
URL: https://github.com/apache/spark/pull/53838

   …romConstraints rule are run as part of the operatorOptimizationRuleSet, so that generated filter on Left leg of Join, in some specific situation, is not missed
   
   ### What changes were proposed in this pull request?
   The change is to merge the batch 
   ```
   Batch("Infer Filters", Once,
           InferFiltersFromGenerate,
           InferFiltersFromConstraints)
   
   ```
   into the batch of rules defined by 
   "operatorOptimizationRuleSet".
   
   ### Why are the changes needed?
   The way current set of optimizer rules work is that 
   ```
   // **step1**
   Batch("Operator Optimization before Inferring Filters", fixedPoint,
   operatorOptimizationRuleSet: _*),
    
   // **step2**
   Batch("Infer Filters", Once,
   InferFiltersFromGenerate,
   InferFiltersFromConstraints),
    
   // **step3**
   Batch("Operator Optimization after Inferring Filters", fixedPoint,
   operatorOptimizationRuleSet: _*)
   ```
   In the batch of rules "operatorOptimizationRuleSet", the conversion of Joins like "Left Outer" to Inner happens.{}
   After that "InferFiltersFromConstraints" is called which is able to create new constraints like IsNotNull, to be pushed on either side of the Inner Join tables.{}
    
   Notice that "operatorOptimizationRuleSet" is called twice, before and after inferring filters.{}
    
   It so happens that in TPCDS Q5, atleast, the conversion of LeftOuter to Inner for one of the Join cases, happens in step3.
    
   But since, there is no further call of InferFiltersFromConstraints, the IsNotNull constraints generation is missed, for the Left Leg of the Join.
   
   Please note that the numerous plan changes are cosmetic ( caused by reordering of text), except in case of q5, where  two new  not null constraints are created for the Left Leg of the Join.
   
   I will try to fix the cosmetic change issue though might be difficult ..
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   Passing of existing tests.. 
   will be adding a dedicated bug test.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3801655387

   The combined PR for SPARK-55185 and SPARK-55072 is
   [PR-SPARK-55072_SPARK-55185](https://github.com/apache/spark/pull/53983)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3761575443

   ## JIRA Issue Information
   
   === Improvement SPARK-55072 ===
   Summary: Inferring new Constraint misses IsNotNull on Left Leg, when an Outer Join gets converted into inner Join
   Assignee: None
   Status: Open
   Affected: ["4.2.0","4.1.1"]
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-4414094218

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #53838: [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) 
URL: https://github.com/apache/spark/pull/53838


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3801702578

   The clean pass for this PR is contingent on the PR for SPARK-55185 ( [PR_SPARK_55185](https://github.com/apache/spark/pull/53968)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on code in PR #53838:
URL: https://github.com/apache/spark/pull/53838#discussion_r2699832388


##########
sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/simplified.txt:
##########
@@ -110,10 +110,11 @@ TakeOrderedAndProject [channel,id,sales,returns,profit]
                                                   InputAdapter
                                                     BroadcastExchange #8
                                                       WholeStageCodegen (14)
-                                                        ColumnarToRow
-                                                          InputAdapter
-                                                            Scan parquet spark_catalog.default.web_returns [wr_item_sk,wr_order_number,wr_return_amt,wr_net_loss,wr_returned_date_sk]
-                                                              ReusedSubquery [d_date_sk] #1
+                                                        Filter [wr_item_sk,wr_order_number]

Review Comment:
   This is the IsNotNull filter which is missed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [WIP][SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3761831365

   There are multiple failures.. especially like ThriftServerQueryTestSuite,  which to me indicates other hidden bugs .. looking into it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55072][SQL]. Ensure IsNotNull inferred filter is not missed for left leg of Inner Join ( obtained via conversion of Left Outer) [spark]

Posted by "ahshahid (via GitHub)" <gi...@apache.org>.
ahshahid commented on PR #53838:
URL: https://github.com/apache/spark/pull/53838#issuecomment-3816247694

   The combined PR [SPARK-55072 + SPARK-55185 + SPARK-55241](https://github.com/apache/spark/pull/53983) is passing, so this PR is ready for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org