You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "theirix (via GitHub)" <gi...@apache.org> on 2025/06/25 07:42:06 UTC
[I] Physical plan pushdown for volatile predicates [datafusion]
theirix opened a new issue, #16545:
URL: https://github.com/apache/datafusion/issues/16545
### Describe the bug
This is a follow-up to a discussion in https://github.com/apache/datafusion/pull/16325#issuecomment-2985522134, which is not directly related to table sampling but could affect it.
I'd like to double-check if a volatile filter pushdown to a Parquet executor is expected. I had implemented the disabling of volatile pushdown filters for a logical plan in #13268. But it seems like the physical optimiser still pushes this predicate to an executor. Should we implement a similar mechanism to make volatile predicates as unsupported filters? In a current physical plan implementation, there is a concept of "unsupported" filters, which can be easily reused for it.
Current behaviour:
Before:
```
[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by LimitedDistinctAggregation:
OutputRequirementExec
ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
FilterExec: random() < 0.1
DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet
```
After:
```
[2025-06-18T18:20:07Z TRACE datafusion::physical_planner] Optimized physical plan by FilterPushdown:
OutputRequirementExec
ProjectionExec: expr=[count(Int64(1))@0 as count(*)]
AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))]
AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))]
DataSourceExec: file_groups={1 group: [[sample.parquet]]}, file_type=parquet, predicate=random() < 0.1
```
### To Reproduce
```sql
set datafusion.execution.parquet.pushdown_filters=true;
create external table data stored as parquet location 'sample.parquet';
SELECT count(*) FROM data WHERE random() < 0.1;
```
### Expected behavior
I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3029427291
@findepi could you please tell if this behaviour makes sense? If not, I could try fixing the physical plan as in #13268, where you have a review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3097890993
> I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.
I am not sure -- does this result in wrong results?
It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3092538385
@alamb @adriangb, given your most recent work on pushdowns, does it appear to be a bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #16545: Physical plan pushdown for volatile predicates
URL: https://github.com/apache/datafusion/issues/16545
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3104952628
> > I expect the physical plan optimiser doesn't perform pushdown of volatile predicates.
>
> I am not sure -- does this result in wrong results?
We don't observe incorrect results (as was the case with the logical optimiser bug when evaluating a predicate twice).
It can produce skewed or non-reproducible results if a volatile predicate evaluates to different values for different row groups (I can create a test to reproduce it).
It could probably affect the table sampling implementation #16325 , which relies on inserting a volatile filter.
> It does make sense in general to have the physical pushdown optimizer follow the same behavior as the logical optimizer
Yes, it is a reasonable assumption about an optimiser.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3104993647
It seems reasonable to me to blacklist some filters. But given that users can create arbitrary trait implementations for PhysicalExpr we obviously can't black list everything. Unless there a volatility method on the trait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Physical plan pushdown for volatile predicates [datafusion]
Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on issue #16545:
URL: https://github.com/apache/datafusion/issues/16545#issuecomment-3137490258
Thank you for the improvement, @adriangb !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org