You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Ted-Jiang (via GitHub)" <gi...@apache.org> on 2023/09/14 04:00:40 UTC
[GitHub] [arrow-datafusion] Ted-Jiang opened a new issue, #7556: Make StatisticsCache share in session level
Ted-Jiang opened a new issue, #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556
### Is your feature request related to a problem or challenge?
In our systems try to pass logical plan to datafusion with enable collect statics. The source table is from remote storage, sometimes it cost a few seconds to read parquet metadata to collect statics.
From log
```rust
datafusion::datasource::listing::table: Not hit cache infer_stats ObjectMeta { location: Path { raw: "working-dir/..-examples-test_case_data-reusemeta-metadata/ddltest/parquet/17d6373b-57ef-f370-34a6-1bd37d156a76/fa2ccb1e-5470-88a2-2fcb-b19779597e96/1/part-00000-f0bfae88-e929-4af1-99be-2599f2b51b3c-c000.snappy.parquet" }, last_modified: 2023-05-18T09:53:04.716427232Z, e_tag: None }, cost 1.5161s
```
So i check the code see there is a cache called `StatisticsCache` construct here:
https://github.com/apache/arrow-datafusion/blob/abea8938b571a4aecddc7185b3acacadcc7dd854/datafusion/core/src/datasource/listing/table.rs#L656
It seems every time build a plan then insert an empty cache, only infer same file statistics in same plan can get benefit.
So I want to share the statics cache in session level 😄 to solve fetch remote file statistics not stable. I think many others query engine did this too.
### Describe the solution you'd like
Add a cache manager to deal with all cache during the session lifetime.
https://github.com/apache/arrow-datafusion/blob/a38480951f40abce7ee2d5919251a1d1607f1dee/datafusion/execution/src/runtime_env.rs#L44-L50
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #7556: Make StatisticsCache share in session level
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1718730756
@alamb @Dandandan @yahoNanJing PTAL Long time not involved in the community 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #7556: Make StatisticsCache share in session level
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1718846726
https://duckdb.org/docs/sql/configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #7556: Make StatisticsCache share in session level
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1720412409
> I think keeping a metadata cache on the RuntimeEnv is reasonable as long as
>
> 1. There is a way to extend / disable the default behavior (as there is with the DiskManager and MemoryPool).
> 2. The default implementation in DataFusion is simple
>
> The rationale for something simple built in but a configurable API is that the exact caching strategy is likely to vary tremendously from system to system (for example, if there is a local file based parquet cache, storing metadata in memory might not make sense, or how to do cache eviction or enforce limits, etc).
>
This suggestion is very important
> Therefore it is unlikely that anything in DataFusion will cover all usecases, so what is built in should be simple and allow users to add whatever specific caching policy they want
>
> Does that makes sense @Ted-Jiang ?
@alamb
Does influx io has the `file statis cache` or the `list files cache` when?
How does influx io resolve the issue that node need to visit the remote storage when generating the execution plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #7556: Make StatisticsCache share in session level
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1718850028
Another bottleneck we found is list thousands files under a remote storage path.
One examples is : One query run in 3 seconds, sometimes it goes to 7 seconds due to long time listing files on remote storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #7556: Make StatisticsCache share in session level
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1720419632
@alamb Thanks for the suggestions ! This make sense.
I will extract a trait like `MemoryPoo`l for user extension their specific caching policy.
As the where should call `get` or `set` in execution: i think there is only one way hard code in physical plan，which means user can not customize where they call get or set cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7556: Make StatisticsCache share in session level
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1720040111
I think keeping a metadata cache on the RuntimeEnv is reasonable as long as
1. There is a way to extend / disable the default behavior (as there is with the DiskManager and MemoryPool).
2. The default implementation in DataFusion is simple
The rationale for something simple built in but a configurable API is that the exact caching strategy is likely to vary tremendously from system to system (for example, if there is a local file based parquet cache, storing metadata in memory might not make sense, or how to do cache eviction or enforce limits, etc).
Therefore it is unlikely that anything in DataFusion will cover all usecases, so what is built in should be simple and allow users to add whatever specific caching policy they want
Does that makes sense @Ted-Jiang ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #7556: Make StatisticsCache share in session level
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1721070992
@liukun4515
> Does influx io has the file statis cache or the list files cache when?
IOx caches (effectively) the list of files and a (very small) subset of the statistics (in our case just the min/max timestamp values). Our metadata catalog (see below) did not have space to store the entire parquet file metadata (with per-row group statistics)
Also, at the moment IOx has an in memory cache of the actual parquet data, which means it effectively always reads the entire objects from storage (though we may change this at some point)
> How does influx io resolve the issue that node need to visit the remote storage when generating the execution plan?
IOx has its own, separate, metadata catalog that stores information about the schema and what files store data for each table as well as what partition (IOx keeps the data segregated in daily partitions typically).
Thus we never use `LIST` operations on object store
THe hgih level architecture is described here: https://www.influxdata.com/blog/influxdb-3-0-system-architecture/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang closed issue #7556: Make StatisticsCache share in session level
Posted by "Ted-Jiang (via GitHub)" <gi...@apache.org>.
Ted-Jiang closed issue #7556: Make StatisticsCache share in session level
URL: https://github.com/apache/arrow-datafusion/issues/7556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #7556: Make StatisticsCache share in session level
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #7556:
URL: https://github.com/apache/arrow-datafusion/issues/7556#issuecomment-1722488428
> @liukun4515
>
> > Does influx io has the file statis cache or the list files cache when?
>
> IOx caches (effectively) the list of files and a (very small) subset of the statistics (in our case just the min/max timestamp values). Our metadata catalog (see below) did not have space to store the entire parquet file metadata (with per-row group statistics)
>
> Also, at the moment IOx has an in memory cache of the actual parquet data, which means it effectively always reads the entire objects from storage (though we may change this at some point)
>
> > How does influx io resolve the issue that node need to visit the remote storage when generating the execution plan?
>
> IOx has its own, separate, metadata catalog that stores information about the schema and what files store data for each table as well as what partition (IOx keeps the data segregated in daily partitions typically).
>
> Thus we never use `LIST` operations on object store
>
> THe hgih level architecture is described here: https://www.influxdata.com/blog/influxdb-3-0-system-architecture/
Thanks for your detailed comments, I will take a look this blogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org