You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "LiaCastaneda (via GitHub)" <gi...@apache.org> on 2026/01/09 16:33:31 UTC

[I] Dynamic Filter marked as not used [datafusion]

LiaCastaneda opened a new issue, #19715:
URL: https://github.com/apache/datafusion/issues/19715

   ### Describe the bug
   
   For one user the `DynamicFilterPhysicalExpr` function is getting marked as unused. I'm openning this issue to discuss why this could be happening .
   More context on https://github.com/apache/datafusion/issues/18566#issuecomment-3728690204
   
   ### To Reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729870328

   I'll note that if the using the Arc pointers, etc. is too much of an issue or we can't resolve this bug / usage we could always go for an explicit API to register "consumers" of a dynamic expression. But I think that'd be a worse user experience (things need to remember to call it, etc.).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed issue #19715: Dynamic Filter marked as not used
URL: https://github.com/apache/datafusion/issues/19715


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733201112

   Maybe we can say it’s used if either one of the outer or inner Arc have > 1 ref?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "tobixdev (via GitHub)" <gi...@apache.org>.
tobixdev commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3732318606

   Thank you for opening the issue @LiaCastaneda ! I am not sure what the problem is but I guess it has something to do with our custom data source as our project is far away from production ready.
   
   For example, we currently do not implement reset state in our custom data source. Maybe that's it.
   
   I think the next step is that I'll create a reproducer. Hopefully, then we can identify and fix the problem. I'll try to do it sometime next week.
   
   Thanks again and I'll be back with a reproducer!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733260529

   (will open a PR shortly)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733259476

   yep that works, if the dynamic filter is used the consumer will have either a ref >1 in the inner struct (because `with_new_children` was called and cloned inner) or outer struct (`with_new_children` was not called), otherwise there is no way for the consumer to see updates on the DF. 
   
   I have a repro similar to @tobixdev `DataSource` [here](https://github.com/LiaCastaneda/datafusion/tree/lia/repro-and-fix) and verified @adriangb suggestion fixes it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733169909

   I suspect this is happening because you have a custom `DataSource`, that differently to `FileScanConfig` (used for parquet) doesn't call `reassign_expr_columns` inside `try_pushdown_filters` ([code](https://github.com/apache/datafusion/blob/458b49109af58e678520edebbad9fb3edfd26992/datafusion/datasource/src/file_scan_config.rs#L802)), so `with_new_children` is never called. We should fix this since it should work out of the box for any DataSource, I will try to think of something...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729914428

   πŸ‘‹ @tobixdev the `inner` struct of `DynamicFilterPhysicalExpr` gets updated (the ref count increases from 1 to 2) when we call `with_new_children` during the filter pushdown optimization rule. 
   
   I'm trying to understand why `with_new_children` is not being called for your query πŸ€”  this might be an edge case I overlooked. The reason why I didn't used the strong count of `DynamicFilterPhysicalExpr` is precisely because when [with_new_children](https://github.com/apache/datafusion/blob/209a0a2e86dda54e7dc00f5715ed1e35ea1af3cb/datafusion/physical-expr/src/expressions/dynamic_filters.rs#L357) is called for `DynamicFilterPhysicalExpr` it creates a new `DynamicFilterPhysicalExpr` but keeps the Arc of the `inner` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Dynamic Filter marked as not used [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729917328

   Could it be that https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.reset_state is being called (or should be called)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org