You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Pajaraja (via GitHub)" <gi...@apache.org> on 2025/05/30 16:12:35 UTC

[PR] [SPARK-52353] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations [spark]

Pajaraja opened a new pull request, #51056:
URL: https://github.com/apache/spark/pull/51056

   ### What changes were proposed in this pull request?
   
   Remove constraints and statistics from LogicalRDDs used to store results of previous iterations.
   
   ### Why are the changes needed?
   
   In some cases logicalRDDs used for memorizing the previous iteration results produces incorrect statistics, so the filter gets pruned out, leading to an infinite recursion. We should fix this bug by removing these stats.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   
   ### How was this patch tested?
   
   The failing test case was added as a golden file. This test requires shutting off the optimization of converting small results into LocalRelation.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52353][SQL] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations [spark]

Posted by "Pajaraja (via GitHub)" <gi...@apache.org>.
Pajaraja closed pull request #51056: [SPARK-52353][SQL] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations
URL: https://github.com/apache/spark/pull/51056


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org