You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/03/31 04:03:04 UTC
[PR] [SPARK-56176][SPARK-56232][SQL] V2-native ANALYZE TABLE/COLUMN with stats propagation to FileScan [spark]
LuciferYang opened a new pull request, #55108:
URL: https://github.com/apache/spark/pull/55108
## What changes were proposed in this pull request?
This PR is part of [SPARK-56170](https://issues.apache.org/jira/browse/SPARK-56170). It implements V2-native ANALYZE TABLE and ANALYZE COLUMN for file tables, removing the last V1 fallbacks in `ResolveSessionCatalog`, and propagates stored statistics through `FileScan` for query optimization.
### 1. V2-native ANALYZE TABLE (`AnalyzeTableExec`)
New physical plan node that computes table statistics and persists them as table properties via `TableCatalog.alterTable(TableChange.setProperty(...))`:
- `spark.sql.statistics.totalSize`: computed from `fileIndex.sizeInBytes`
- `spark.sql.statistics.numRows`: computed from `df.count()` (skipped when `NOSCAN`)
### 2. V2-native ANALYZE COLUMN (`AnalyzeColumnExec`)
New physical plan node that computes column-level statistics using `CommandUtils.computeColumnStats` and persists them as table properties:
- `spark.sql.statistics.colStats.<col>.<stat>`: min, max, nullCount, distinctCount, avgLen, maxLen
- Supports both `FOR COLUMNS` and `FOR ALL COLUMNS` syntax
### 3. Stats propagation to FileScan
- `FileScan.numRows()`: reads stored row count from `__numRows` option (set by ANALYZE TABLE)
- `FileTable.mergedOptions`: injects `__numRows` from `catalogTable.stats.rowCount` into scan options so `FileScan.estimateStatistics()` can use it
### 4. V1 fallback removal
Removes the AnalyzeTable and AnalyzeColumn FileTable cases from `ResolveSessionCatalog` — these are now handled natively by `DataSourceV2Strategy` routing to `AnalyzeTableExec`/`AnalyzeColumnExec`.
## Why are the changes needed?
Previously, ANALYZE TABLE/COLUMN for V2 file tables was routed through V1 commands (`AnalyzeTableCommand`/`AnalyzeColumnCommand`) via `ResolveSessionCatalog` fallbacks. This broke the V2 design principle where table operations should go through the V2 catalog API. The V1 path also stored stats only in the Hive metastore, making them invisible to V2 scan planning.
With this change:
- Stats are stored as table properties via the standard `TableCatalog.alterTable()` API
- `FileScan` can read stored row counts for better query optimization
- No V1 command routing needed for ANALYZE operations
## Does this PR introduce _any_ user-facing change?
No. `ANALYZE TABLE` and `ANALYZE TABLE FOR COLUMNS` produce the same results. Statistics are stored in the catalog and used for query optimization as before.
### How was this patch tested?
- Pass Github Actions
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56176][SPARK-56232][SQL] V2-native ANALYZE TABLE/COLUMN with stats propagation to FileScan [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55108:
URL: https://github.com/apache/spark/pull/55108#issuecomment-4159717910
This is the 4th pr in this series, and it is based on https://github.com/apache/spark/pull/55091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56176][SQL] V2-native ANALYZE TABLE/COLUMN with stats propagation to FileScan [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55108:
URL: https://github.com/apache/spark/pull/55108#issuecomment-5013557404
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56176][SQL] V2-native ANALYZE TABLE/COLUMN with stats propagation to FileScan [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55108: [SPARK-56176][SQL] V2-native ANALYZE TABLE/COLUMN with stats propagation to FileScan
URL: https://github.com/apache/spark/pull/55108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org