You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/02 20:07:48 UTC
[I] Add limit to `DefaultFileStatisticsCache` [datafusion]
alamb opened a new issue, #19052:
URL: https://github.com/apache/datafusion/issues/19052
Also `listing file statistics cache` seems to not have any memory limit unlike `metadata cache` for example.
Is that by design , do you think we need to add similar limit for this cache too ?
_Originally posted by @bharath-techie in https://github.com/apache/datafusion/issues/18971#issuecomment-3600061959_
Basically the cache used in ListingTable comes from here:
https://github.com/apache/datafusion/blob/81512da2b0aaa474f6c4ba205b05eea7b3095176/datafusion/core/src/datasource/listing_table_factory.rs#L188-L187
Which somewhat unobviously sets a `DefaultFileStatisticsCache` here
https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/datafusion/catalog-listing/src/table.rs#L260-L259
The `DefaultFileStatisticsCache` has no limit:
https://github.com/apache/datafusion/blob/7d8b8602ad1be2f61f6a8ebb253ace9d85304ea7/datafusion/execution/src/cache/cache_unit.rs#L41-L40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #19052: Add limit to `DefaultFileStatisticsCache`
URL: https://github.com/apache/datafusion/issues/19052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3603814383
It turns out @alchemist51 also pointed this out in
- https://github.com/apache/datafusion/pull/18855/files#r2560294003
I didn't understand at first what was meant
> A little bit side topic. @alamb was looking into other caches, didn't understand why DefaultFileStatisticsCache isn't memory bound yet. Do we want to do it incrementally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3637827742
Thank you @jizezhang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3603796097
My suggestion would be to implement @nuno-faria 's ideas here: https://github.com/apache/datafusion/issues/18953#issuecomment-3598406492
And then once we had a real trait, add the ability to track memory used by Statistics and then use that measurement to implement a size limited cache
This would be similar in design to @BlakeOrth 's PR here
- https://github.com/apache/datafusion/pull/18855
I suggest a sequence of smaller PRs:
1. Add a method to statistics like `Statistics::heap_size` that reports the heap (allocation) size for statistics -- following the semantics of https://github.com/apache/arrow-rs/blob/b93fa52e47a83dff130709a6e80a4be5017dbf09/parquet/src/file/metadata/memory.rs#L35
2. Update the `DefaultFileStatisticsCache` to have a memory limit and use the heap_size from 1 to constrain the size
3. Add a config parameter , perhaps `datafusion.runtime.statistics_cache_limit`, similar to [`datafusion.runtime.metadata_cache_limit`](https://datafusion.apache.org/user-guide/configs.html#runtime-configuration-settings), and set the default statistics cache in https://github.com/apache/datafusion/blob/71fcd03d614c466fdc8820084dc2896f69775f2b/datafusion/execution/src/cache/cache_manager.rs#L140-L141 (following the example of `DefaultFilesMetadataCache`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3633369401
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "abhita (via GitHub)" <gi...@apache.org>.
abhita commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3805711175
@mkleen @alamb
As the plan here is to enforce limit on `DefaultFileStatisticsCache` , are we also planning on any mechanisms to handle stale entries here? perhaps through eviction or TTL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3637726808
@mkleen Could you try `take` this issue if you still plan to work on it? It looks like I cannot unassign myself...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3633634227
Dear @jizezhang I am already working on https://github.com/apache/datafusion/issues/19217, and this ticket is basically a requirement for it. I was wondering if it would be ok with you if I take it over. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3634292773
Hey @mkleen feel free to take it if that makes things easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3811479778
> [@mkleen](https://github.com/mkleen) [@alamb](https://github.com/alamb) As the plan here is to enforce limit on `DefaultFileStatisticsCache` , are we also planning on any mechanisms to handle stale entries here? perhaps through eviction or TTL?
@abhita Currently it's handled by eviction see https://github.com/apache/datafusion/pull/20047 for details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3700940902
@mkleen and @jizezhang is there any update on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3703724541
> > [@mkleen](https://github.com/mkleen) and [@jizezhang](https://github.com/jizezhang) is there any update on this issue?
>
> Yes, I am working on it as part of [#19217](https://github.com/apache/datafusion/issues/19217). We would like to have this for the next release, right?
Yes, please, let's try and get it done for 52.0.0 if possible (we are starting to test next week I believe)
If you can break the work into a few smaller PRs I will be more likely to be able to help review it quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3701473762
> [@mkleen](https://github.com/mkleen) and [@jizezhang](https://github.com/jizezhang) is there any update on this issue?
Yes, I am working on it as part of https://github.com/apache/datafusion/issues/19217. We would like to have this for the next release, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3633665441
However, I guess it's ok, we can merge it all together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add limit to `DefaultFileStatisticsCache` [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #19052:
URL: https://github.com/apache/datafusion/issues/19052#issuecomment-3637734138
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org