You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/04/13 07:17:19 UTC
[PR] [SPARK-56337][SQL] Propagate analyzed sizeInBytes to V2 FileScan estimates [spark]
LuciferYang opened a new pull request, #55322:
URL: https://github.com/apache/spark/pull/55322
### What changes were proposed in this pull request?
Extend SPARK-56176's V2-native ANALYZE TABLE stats propagation pattern by
adding `FileTable.SIZE_IN_BYTES_KEY` alongside the existing `NUM_ROWS_KEY`,
and teaching `FileScan.estimateStatistics().sizeInBytes()` to prefer the
catalog-injected size over the file-listing-based estimate.
- `FileTable.mergedOptions` now injects `CatalogTable.stats.sizeInBytes`
(when present) into the scan-time options under `SIZE_IN_BYTES_KEY`,
alongside the `NUM_ROWS_KEY` injection added by SPARK-56176.
- `FileScan` exposes a new `storedSizeInBytes` helper that mirrors
`storedNumRows`. `estimateStatistics()` prefers it over the existing
file-index-based estimate when present, falling back to the listing-based
computation when absent.
### Why are the changes needed?
SPARK-56176 introduced `numRows` propagation but did not cover `sizeInBytes`
for non-partitioned catalog file source tables:
- Partitioned catalog tables that use `useCatalogFileIndex` already see the
analyzed `sizeInBytes` because `CatalogFileIndex.sizeInBytes` returns the
catalog-supplied value directly.
- Non-partitioned catalog tables use `InMemoryFileIndex`, whose `sizeInBytes`
is computed from file listing rather than from `CatalogTable.stats`.
The result is that `ANALYZE TABLE COMPUTE STATISTICS` on a non-partitioned V2
file catalog table populates `CatalogTable.stats.sizeInBytes` but the V2 scan
keeps using the listing-based estimate, so CBO decisions (broadcast join
threshold, filter selectivity) silently diverge from V1 and from partitioned
V2 tables. This change closes that gap symmetrically with how SPARK-56176
closed the `numRows` gap.
### Does this PR introduce _any_ user-facing change?
Yes (CBO-affecting). For non-partitioned V2 file source catalog tables that
have analyzed stats, the V2 scan's `sizeInBytes` estimate now matches the
analyzed value. Workloads that rely on `ANALYZE TABLE` to tune broadcast
thresholds against non-partitioned V2 catalog tables will see the same plan
shapes V1 produced.
### How was this patch tested?
New `V2FileScanSizeInBytesSuite` (3 tests):
- `Analyzed sizeInBytes flows into V2 scan estimate (non-partitioned)`:
populates `CatalogTable.stats` directly via `alterTableStats` (sidesteps
the ANALYZE-via-table-properties bridge that requires `HiveExternalCatalog`)
and asserts the V2 scan's optimized-plan stats reflect the analyzed value.
- `Analyzed sizeInBytes flows into V2 scan estimate (partitioned)`:
guards that the partitioned path keeps producing analyzed stats end to end
(no regression vs `CatalogFileIndex.sizeInBytes`).
- `Without ANALYZE TABLE, sizeInBytes falls back to file-listing estimate`:
guards the fallback path: when `CatalogTable.stats` is empty, the V2 scan
produces a positive listing-based estimate (no zero-or-default regression).
Existing suites verified passing:
- `FileMetadataColumnsV2Suite` (24 tests, SPARK-56335)
- `ParquetMetadataRowIndexV2Suite` (9 tests, SPARK-56371)
Total: 36 tests across the stacked-branch suites. Scalastyle clean.
### Dependencies
Builds on top of SPARK-56371 (row_index metadata for V2 Parquet).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56337][SQL] Propagate analyzed sizeInBytes to V2 FileScan estimates [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55322:
URL: https://github.com/apache/spark/pull/55322#issuecomment-5053095616
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56337][SQL] Propagate analyzed sizeInBytes to V2 FileScan estimates [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55322: [SPARK-56337][SQL] Propagate analyzed sizeInBytes to V2 FileScan estimates
URL: https://github.com/apache/spark/pull/55322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org