You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/03/25 20:21:41 UTC

[I] Avoid evaluating filters when they can be discarded purely from statistics [datafusion]

adriangb opened a new issue, #15425:
URL: https://github.com/apache/datafusion/issues/15425

   ### Is your feature request related to a problem or challenge?
   
   Currently stats filter pruning (both at the row group and page level) has one of two outcomes per container:
   1. This container cannot possibly match the filter (discard it).
   2. This container *may* match the filter, but which rows to include or exclude needs to be confirmed by evaluating each row of the data.
   
   There is a big optimization here which is *if we know that every row in the container matches the filter, we don't need to evaluate the filter at all*.
   
   Consider a column `name` with values `["Adrian", "Adrian", "Adrian"]`. The min/max stats are `"Adrian"/"Adrian"`. A query with the filter `name = "Adrian"` should not need to ever read the column to know that all rows match the filter.
   
   Another relevant case is a `ts` column with values `["2025-01-01T00:00:00Z", ..., "2025-01-01T00:01:32Z"]`. The values need not be sorted or ordered, but let's say that the min/max stats are `"2025-01-01T00:00:00Z"/"2025-01-01T00:01:32Z"`. For a filter `ts > '2024-12-31T00:00:00Z'` there should be no need to evaluate the filter on every row: we know just from stats that every row matches.
   
   We could incorporate this change, but it would require some refactoring of https://github.com/apache/datafusion/blob/main/datafusion/physical-optimizer/src/pruning.rs and consumers.
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Avoid evaluating filters when they can be discarded purely from statistics [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #15425:
URL: https://github.com/apache/datafusion/issues/15425#issuecomment-2752626326

   I think that has more to do with the `datafusion.execution.collect_statistics` setting and use of those statistics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Avoid evaluating filters when they can be discarded purely from statistics [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #15425:
URL: https://github.com/apache/datafusion/issues/15425#issuecomment-3649725049

   Looping back to this: @matthewmturner I think I now understand your optimization proposal better. I think what you are suggesting may be even easier to implement than this is. It could be something similar to https://github.com/apache/datafusion/pull/19136 but instead looking for `max(<col>)`/`min(<col>)` and replacing it with values from the file-level statistics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Avoid evaluating filters when they can be discarded purely from statistics [datafusion]

Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #15425:
URL: https://github.com/apache/datafusion/issues/15425#issuecomment-2752619613

   I think this is similar to something I recently asked on Discord - except I had in mind using only the metadata stats for queries like "SELECT MAX(timestamp) FROM quotes"
   
   This was my full comment / question
   
   "Im doing some data exploration on a table in datafusion where im running the following `SELECT MAX(timestamp) FROM quotes`.  The `quotes` table is about 100GB of data.   When i run `EXPLAIN ANALYZE` on this plan i see from the `ParquetExec` 6B+ output rows and 30GB+ of bytes scanned.  Given that I'm only getting the MAX for the column shouldnt I be able to get this by doing much less work and only looking at the row group metadata stats and not scanning any data?  That would give me a huge performance improvement (the metadata load time is < 1% of the total time scanning)."


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Avoid evaluating filters when they can be discarded purely from statistics [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #15425:
URL: https://github.com/apache/datafusion/issues/15425#issuecomment-2752634913

   Well if you did set `datafusion.execution.collect_statistics` I _think_ those stats would be used to calculate `max` _but_ I think you're right that even if you didn't collect statistics upfront if that expression could be pushed down into each individual file scan then it could be optimized. Maybe related to https://github.com/apache/datafusion/issues/14993? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org