You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "LiaCastaneda (via GitHub)" <gi...@apache.org> on 2025/09/09 11:07:17 UTC
[I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
LiaCastaneda opened a new issue, #17486:
URL: https://github.com/apache/datafusion/issues/17486
### Describe the bug
Hello! We are very interested in using the Dynamic Filtering on hash joins optim feature, so I brought the dynamic filtering work https://github.com/DataDog/datafusion/pull/43 to test it on a production environment.
The API works as expected (we do receive dynamic filters). However the change put all our resources at maximum CPU and profiling showed that almost 2/3 of the overall CPU usage was spent in `min_batch` and `max_batch` when comparing **lists**. Specifically the profiler showed the issue was in min_batch -> min_max_batch_generic -> ScalarValue::partial_cmp for lists (via [partial_cmp_list](https://github.com/apache/datafusion/blob/c83466c20ea5ec2c2c98e61917a3324cd71211ee/datafusion/common/src/scalar/mod.rs?utm_source=chatgpt.com#L528)).
It looks like comparing list values during bounds computation is expensive, is this expected?
Also I noticed `compute_bounds` is called for every query that has Hash Join by default.
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3270984856
> I wonder if it would make sense to compute dynamic filters only if/when there is a node that consumes/ asks for the DynamicFilterPhysicalExpr ? maybe while polling the right side?
It certainly makes sense to me to avoid doing work for DynamicFilters that are never used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3280262788
> There are a couple of other things I noticed while using the API that I think are worth mentioning, but I can file them as separate issues. I’m also happy to take a stab at any of those if no one else is interested!
Please do! Happy to also do a quick call to discuss.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3279886885
> if the physical optimizer doesn't make a reasonable join order, which means, the build side could be huge, the if we do computation for dynamic filter, things will be bad.
Do you think we need to file an issue for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3279754503
Amazing! Shall we close this then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3279790910
> > [@LiaCastaneda](https://github.com/LiaCastaneda) could you try this on main now? I wonder if [#17444](https://github.com/apache/datafusion/pull/17444) made a difference.
>
> I can confirm it mitigates the CPU issue :)
I suspect this is because the old code always created a new ScalarValue for the min/max, even if the min/max didn't change. For ListArrays specifically this is quite expensive. The new code only makes a new Scalarvalue when the min/max changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda closed issue #17486: High CPU during dynamic filter bound computation: min_batch/max_batch
URL: https://github.com/apache/datafusion/issues/17486
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3270213815
cc @adriangb -- sorry for the direct ping. Since you did most of the dynamic filtering work, do you know if this behavior is expected? 🙇♀️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3279760485
yes, thanks for the fix! (even if it wasn’t meant to be one 😄 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3280252533
There are a couple of other things I noticed while using the API that I think are worth mentioning, but I can file them as separate issues. I’m also happy to take a stab at any of those if no one else is interested!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3275223813
Will give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3270950177
Disabling `optimizer.enable_dynamic_filter_pushdown ` makes the issue go away, since the bounds are computed for every query containing hash join if the option is enabled. I wonder if it would make sense to compute dynamic filters only if/when there is a node that consumes/ asks for the`DynamicFilterPhysicalExpr` ? maybe while polling the right side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3275158766
> I agree finding some way to avoid making a bad join order worse is important
We don't pass the DataFusion logical optimizer to our logical plans, choosing the best join ordering is something we already account for when building the substrait plan, but I think it would be nice to have as an optim rule maybe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3274003009
I'm worried about the scenario: if the physical optimizer doesn't make a reasonable join order, which means, the build side could be huge, the if we do computation for dynamic filter, things will be bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3274712288
I think the key will be to minimize the overhead that computing the dynamic filters add to the join
I agree finding some way to avoid making a bad join order worse is important
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3279741137
> @LiaCastaneda could you try this on main now? I wonder if https://github.com/apache/datafusion/pull/17444 made a difference.
I can confirm it mitigates the CPU issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3270666925
> It looks like comparing list values during bounds computation is expensive, is this expected?
I strongly suspect the comparison of lists and other structured scalars could be improved significantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3271707155
> > I wonder if it would make sense to compute dynamic filters only if/when there is a node that consumes/ asks for the DynamicFilterPhysicalExpr ? maybe while polling the right side?
>
> It certainly makes sense to me to avoid doing work for DynamicFilters that are never used
We could add some notion of "subscribers" to a dynamic filter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3275160952
> I agree finding some way to avoid making a bad join order worse is important
We don't pass the DataFusion logical optimizer to our logical plans, choosing the best join ordering is something we already account for when building the substrait plan, but I think it would be nice to have as an optim rule maybe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] High CPU during dynamic filter bound computation: min_batch/max_batch [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #17486:
URL: https://github.com/apache/datafusion/issues/17486#issuecomment-3275183333
@LiaCastaneda could you try this on main now? I wonder if https://github.com/apache/datafusion/pull/17444 made a difference. I'd be surprised if this is slow now: it's just tracking a min and max value for each partition (so doing a binary compare to another scalar for each row on the build side) vs. the already existing work of building a hash table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org