You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "LiaCastaneda (via GitHub)" <gi...@apache.org> on 2026/01/09 16:33:31 UTC
[I] Dynamic Filter marked as not used [datafusion]
LiaCastaneda opened a new issue, #19715:
URL: https://github.com/apache/datafusion/issues/19715
### Describe the bug
For one user the `DynamicFilterPhysicalExpr` function is getting marked as unused. I'm openning this issue to discuss why this could be happening .
More context on https://github.com/apache/datafusion/issues/18566#issuecomment-3728690204
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729870328
I'll note that if the using the Arc pointers, etc. is too much of an issue or we can't resolve this bug / usage we could always go for an explicit API to register "consumers" of a dynamic expression. But I think that'd be a worse user experience (things need to remember to call it, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed issue #19715: Dynamic Filter marked as not used
URL: https://github.com/apache/datafusion/issues/19715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733201112
Maybe we can say itβs used if either one of the outer or inner Arc have > 1 ref?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "tobixdev (via GitHub)" <gi...@apache.org>.
tobixdev commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3732318606
Thank you for opening the issue @LiaCastaneda ! I am not sure what the problem is but I guess it has something to do with our custom data source as our project is far away from production ready.
For example, we currently do not implement reset state in our custom data source. Maybe that's it.
I think the next step is that I'll create a reproducer. Hopefully, then we can identify and fix the problem. I'll try to do it sometime next week.
Thanks again and I'll be back with a reproducer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733260529
(will open a PR shortly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733259476
yep that works, if the dynamic filter is used the consumer will have either a ref >1 in the inner struct (because `with_new_children` was called and cloned inner) or outer struct (`with_new_children` was not called), otherwise there is no way for the consumer to see updates on the DF.
I have a repro similar to @tobixdev `DataSource` [here](https://github.com/LiaCastaneda/datafusion/tree/lia/repro-and-fix) and verified @adriangb suggestion fixes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3733169909
I suspect this is happening because you have a custom `DataSource`, that differently to `FileScanConfig` (used for parquet) doesn't call `reassign_expr_columns` inside `try_pushdown_filters` ([code](https://github.com/apache/datafusion/blob/458b49109af58e678520edebbad9fb3edfd26992/datafusion/datasource/src/file_scan_config.rs#L802)), so `with_new_children` is never called. We should fix this since it should work out of the box for any DataSource, I will try to think of something...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729914428
π @tobixdev the `inner` struct of `DynamicFilterPhysicalExpr` gets updated (the ref count increases from 1 to 2) when we call `with_new_children` during the filter pushdown optimization rule.
I'm trying to understand why `with_new_children` is not being called for your query π€ this might be an edge case I overlooked. The reason why I didn't used the strong count of `DynamicFilterPhysicalExpr` is precisely because when [with_new_children](https://github.com/apache/datafusion/blob/209a0a2e86dda54e7dc00f5715ed1e35ea1af3cb/datafusion/physical-expr/src/expressions/dynamic_filters.rs#L357) is called for `DynamicFilterPhysicalExpr` it creates a new `DynamicFilterPhysicalExpr` but keeps the Arc of the `inner`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Dynamic Filter marked as not used [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19715:
URL: https://github.com/apache/datafusion/issues/19715#issuecomment-3729917328
Could it be that https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.reset_state is being called (or should be called)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org