You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/05/09 05:12:34 UTC
[I] Pass `PartitionedFile` into `FileSource` for late file stats based pruning [datafusion]
adriangb opened a new issue, #16000:
URL: https://github.com/apache/datafusion/issues/16000
### Is your feature request related to a problem or challenge?
As we continue to make progress landing dynamic filters it opens up the opportunity for new optimizations.
This one deals with late evaluation of file-level statistics.
In particular, we *may* have file level statics available at planning time (see `datafusion.execution.collect_statistics` - our system does a similar thing in a different way).
Before dynamic filters there was no point in re-evaluating these right before scanning a file but now it's possible that e.g. a TopK operator passed down a `ts > '2025-05-08T00:00:00Z'` filter -> we may be able to exclude the entire file based on this filter + file level statistics -> we avoid reading any Parquet metadata, etc.
In particular, change:
```rust
trait FileSource {
fn open(file_meta: FileMeta) ...
```
To:
```rust
trait FileSource {
fn open(file_meta: FileMeta, file: &PartitionedFile) ...
```
And call it as so from here:
https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/datasource/src/file_stream.rs#L129
Then we can implement [PruningStatitics](https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/physical-optimizer/src/pruning.rs#L79) for [Statistics](https://github.com/pydantic/datafusion/blob/649851d59cdac80fcae51d66f82f1b47d2aaa3b4/datafusion/common/src/stats.rs#L270) et. voilá!
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Pass `PartitionedFile` into `FileSource` for late file stats based pruning [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #16000:
URL: https://github.com/apache/datafusion/issues/16000#issuecomment-2868157070
POC at https://github.com/apache/datafusion/pull/16014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Pass `PartitionedFile` into `FileSource` for late file stats based pruning [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #16000:
URL: https://github.com/apache/datafusion/issues/16000#issuecomment-2865140566
cc @berkaysynnada @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Pass `PartitionedFile` into `FileSource` for late file stats based pruning [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed issue #16000: Pass `PartitionedFile` into `FileSource` for late file stats based pruning
URL: https://github.com/apache/datafusion/issues/16000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Pass `PartitionedFile` into `FileSource` for late file stats based pruning [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #16000:
URL: https://github.com/apache/datafusion/issues/16000#issuecomment-2867822624
I think the idea of passing the partitioned file (with the optional per-file statistics) is an excellent idea 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org