You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "haohuaijin (via GitHub)" <gi...@apache.org> on 2026/02/02 12:32:09 UTC

[PR] fix: filter pushdown when merge filter [datafusion]

haohuaijin opened a new pull request, #20110:
URL: https://github.com/apache/datafusion/pull/20110

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes #.
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #20110:
URL: https://github.com/apache/datafusion/pull/20110


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3841991050

   No worries. Let's merge this once CI passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3847348357

   @hengfeiyang we can certainly put this on the docket for the next minor release, but we don’t tend to do more than 1 minor release per major version, so this may have to wait until the next major version if there aren’t many other fixes. Would you be able to point your system at a git commit or fork with this commit cherry picked? That’s what we do


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3838741346

   cc @adriangb, ready for reviews


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "hengfeiyang (via GitHub)" <gi...@apache.org>.
hengfeiyang commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3844937753

   Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3835320709

   @haohuaijin please ping me when this is ready for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3843766200

   🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#discussion_r2759202862


##########
datafusion/physical-plan/src/filter.rs:
##########
@@ -623,10 +623,26 @@ impl ExecutionPlan for FilterExec {
             return Ok(FilterPushdownPropagation::if_all(child_pushdown_result));
         }
         // We absorb any parent filters that were not handled by our children
-        let unsupported_parent_filters =
-            child_pushdown_result.parent_filters.iter().filter_map(|f| {
-                matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
-            });
+        let mut unsupported_parent_filters: Vec<Arc<dyn PhysicalExpr>> =
+            child_pushdown_result
+                .parent_filters
+                .iter()
+                .filter_map(|f| {
+                    matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
+                })
+                .collect();

Review Comment:
   Note: this is the same code, just added the `mut` which caused a reformat.



##########
datafusion/physical-plan/src/filter.rs:
##########
@@ -623,10 +623,26 @@ impl ExecutionPlan for FilterExec {
             return Ok(FilterPushdownPropagation::if_all(child_pushdown_result));
         }
         // We absorb any parent filters that were not handled by our children
-        let unsupported_parent_filters =
-            child_pushdown_result.parent_filters.iter().filter_map(|f| {
-                matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
-            });
+        let mut unsupported_parent_filters: Vec<Arc<dyn PhysicalExpr>> =
+            child_pushdown_result
+                .parent_filters
+                .iter()
+                .filter_map(|f| {
+                    matches!(f.all(), PushedDown::No).then_some(Arc::clone(&f.filter))
+                })
+                .collect();
+
+        // If this FilterExec has a projection, the unsupported parent filters
+        // are in the output schema (after projection) coordinates. We need to
+        // remap them to the input schema coordinates before combining with self filters.
+        if self.projection.is_some() {

Review Comment:
   👍🏻 note that this is because filters get evaluated before the projection



##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -3720,3 +3720,90 @@ async fn test_hashjoin_dynamic_filter_pushdown_is_used() {
         );
     }
 }
+
+/// related to https://github.com/apache/datafusion/issues/20109
+#[tokio::test]
+async fn test_filter_with_projection_pushdown() {

Review Comment:
   Any chance you could add an SLT test that reproduces this? It could go in datafusion/sqllogictest/test_files/parquet_filter_pushdown.slt



##########
datafusion/core/tests/physical_optimizer/filter_pushdown.rs:
##########
@@ -3720,3 +3720,90 @@ async fn test_hashjoin_dynamic_filter_pushdown_is_used() {
         );
     }
 }
+
+/// related to https://github.com/apache/datafusion/issues/20109

Review Comment:
   ```suggestion
   /// Regression test for https://github.com/apache/datafusion/issues/20109
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "hengfeiyang (via GitHub)" <gi...@apache.org>.
hengfeiyang commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3847412097

   Yes. definitely we can do that, actually we did that in the past if we can't get a release soon. just we don't like to use fork version if we can have official release. 
   
   Thank you for you reply. we will use fork version before next release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3849938501

   (BTW I think I may also have a bug in 52 that I'll file tomorrow which might be good to fix too). I'll post here if I do


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3849934894

   > Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.
   
   I am happy to help support another release (run the voting, etc) if someone else is willing do make the tickets, backport and changelog PRs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3883925915

   > > Hi @alamb, Is it possible to release version 52.2.0? This issue has affected our online business, and we have temporarily disabled this feature. It would be great if a minor version could be released quickly.
   > 
   > I am happy to help support another release (run the voting, etc) if someone else is willing do make the tickets, backport and changelog PRs
   
   Update: Filed a ticket to track the next release
    - https://github.com/apache/datafusion/issues/20287


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: filter pushdown when merge filter [datafusion]

Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #20110:
URL: https://github.com/apache/datafusion/pull/20110#issuecomment-3841712116

   Thanks for you reviews @adriangb, i try add a SLT test, but use sql is hard to do.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org