You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonathanc-n (via GitHub)" <gi...@apache.org> on 2026/03/10 17:27:07 UTC
[I] Implement `ANALYZE TABLE` support [datafusion]
jonathanc-n opened a new issue, #20852:
URL: https://github.com/apache/datafusion/issues/20852
### Is your feature request related to a problem or challenge?
ANALYZE TABLE command is used to calculate statistics. Spark's documentation can be seen [here](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-analyze-table.html)
This can allow us to do things like:
- Accurately calculating distinct values (with the choice of just counting everything or using a hyperloglog)
- Allowing the user to recalculate statistics and calculate them upfront instead of lazily if suspected statistics have gotten stale.
- Memtable stat calculations
### Describe the solution you'd like
Something similar to spark's implementation makes sense
Syntax:
```sql
ANALYZE TABLE table_identifier
COMPUTE STATISTICS [ NOSCAN | FOR COLUMNS col [, ...] | FOR ALL COLUMNS ]
```
`COMPUTE STATISTICS` (nothing else after it): Only calculate table stats (num_rows + total_byte_size)
`NOSCAN`: Don't probe/scan any values - just return
`FOR COLUMNS col [, ...]`: table stats + calculating specified columns stats (min, max, null, etc.)
`FOR ALL COLUMNS`: All column stats are calculated (min, max, null, etc.)
- when specified for specific columns; when it is under a certain amount of columns + row count. A direct computation of distinct count can be done.
- Maybe histogram support, this will need to be discussed if anything
- Maybe a way to persist statistics
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement `ANALYZE TABLE` support [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #20852:
URL: https://github.com/apache/datafusion/issues/20852#issuecomment-4033186105
@asolimando wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement `ANALYZE TABLE` support [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #20852:
URL: https://github.com/apache/datafusion/issues/20852#issuecomment-4033185543
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement `ANALYZE TABLE` support [datafusion]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on issue #20852:
URL: https://github.com/apache/datafusion/issues/20852#issuecomment-4033858942
It would be great to have a way to compute statistics programmatically, but the price of computing them generally pays off only if amortized over multiple runs, which requires persisting them.
So unless we have a way to do that, the benefit would be limited.
One possible use-case, not requiring persistence, is to enable query replay with this command activated on some sub-sample of production traffic, note the runtime difference, to easily discover good candidates for statistics pre-computation.
In that space, a more standard technique for discovering candidates is to look at the query catalog/history: for NDV, columns appearing in join conditions, `GROUP BY` clauses, and `DISTINCT` operations are natural candidates.
On the persistence side, for Parquet-backed tables, one option is to embed computed statistics directly in the files using the user-defined Parquet indexes technique: https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/
Stats travel with the data, no external store needed, and other readers simply ignore the extra bytes. Though this wouldn't cover non-Parquet sources and it would require write permission, but it would be a start.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org