You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/09 20:11:59 UTC
[GitHub] [arrow-datafusion] isidentical opened a new issue, #3774: Provide a way to enable source level statistics for tables registered in the CLI
isidentical opened a new issue, #3774:
URL: https://github.com/apache/arrow-datafusion/issues/3774
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
#1347 enabled collection of statistics by default on the `ListingOptions` constructor, though the tables created with `CREATE EXTERNAL TABLE` can't still use this feature since they are created manually.
https://github.com/apache/arrow-datafusion/blob/e54110fb592e03704da5f6ebd832b8fe1c51123b/datafusion/core/src/execution/context.rs#L486-L488
**Describe the solution you'd like**
We already have a per file extension listing option implementation for the `read_` dataframe APIs (e.g. `CsvReadOptions`, `ParquetReadOptions`) and they have sane defaults (like `collect_stats` is `false` for CSV and `true` for Parquet). I wonder whether we can just use them here and obtain the `ListingOptions` directly from them.
**Describe alternatives you've considered**
Leaving as is, or enabling them globally (instead of refactoring that part to use `ReadOptions`) by just setting the flag to true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Provide a way to enable source level statistics for tables registered in the CLI [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #3774:
URL: https://github.com/apache/datafusion/issues/3774#issuecomment-2888721001
I think this is now solved. There is a config option, that can be set in datafusion-cli, to collect stats or not during planning. @Dandandan @isidentical can we close the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #3774: Provide a way to enable source level statistics for tables registered in the CLI
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #3774:
URL: https://github.com/apache/arrow-datafusion/issues/3774#issuecomment-1279925672
I wonder if it ever can be enabled by default for parquet datasets.
The downside for parquet is that when using remote object storage, collecting of statistics takes quite a bit of IO, slowing down simple queries.
I guess at some point we have to switch testing with Delta Lake or Apache Iceberg I guess :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org