You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/04/13 07:04:53 UTC
[PR] [SPARK-56335][SQL] Implement SupportsMetadataColumns in FileTable [spark]
LuciferYang opened a new pull request, #55320:
URL: https://github.com/apache/spark/pull/55320
### What changes were proposed in this pull request?
Expose the V1-compatible `_metadata` struct column (`file_path`, `file_name`,
`file_size`, `file_block_start`, `file_block_length`, `file_modification_time`)
on V2 file-based tables so that queries like
`SELECT _metadata.file_path FROM parquet.`<path>`` work against the V2 scan
path instead of forcing a V1 fallback.
Key changes:
1. **`FileTable` implements `SupportsMetadataColumns`**: `metadataColumns()`
returns a single `_metadata` struct column whose fields come from
`FileFormat.BASE_METADATA_FIELDS`. Formats can extend `metadataSchemaFields`
to expose additional fields (e.g., Parquet's `row_index`, tracked in
SPARK-56371).
2. **`FileScanBuilder.pruneColumns`**: Intercepts the `_metadata` field from
the required schema, stores the pruned metadata struct on
`requestedMetadataFields`, and keeps it out of `readDataSchema` so the
format-specific reader stays unchanged.
3. **`FileScan.readSchema`**: Re-exposes `_metadata` as a trailing struct field
when metadata is requested, so `V2ScanRelationPushDown` can rebind the
downstream attribute reference back to the scan output.
4. **`MetadataAppendingFilePartitionReaderFactory`** (new): Wraps the
format-specific reader factory and appends a single `_metadata` struct value
(via `JoinedRow` + an inner `GenericInternalRow`) to each row. Columnar
reads are disabled while metadata is requested since `ConstantColumnVector`
is scalar and cannot represent a struct column; queries fall back to the row
path.
5. **All six concrete scans** (Parquet/ORC/CSV/JSON/Text/Avro): Take
`requestedMetadataFields` as a trailing default-valued case-class parameter
and call the new `wrapWithMetadataIfNeeded` helper when constructing their
reader factory. Their `ScanBuilder.build()` implementations pass the field
through from `FileScanBuilder`.
Parquet's generated `row_index` metadata field is intentionally out of scope;
follow-up work is tracked in SPARK-56371.
### Why are the changes needed?
Before this change, `_metadata` on a DSv2 file table was unresolvable and the
query fell back to the V1 `FileSourceScanExec` path. This is one of the
remaining blockers for deprecating the V1 file sources (SPARK-56170).
### Does this PR introduce _any_ user-facing change?
Yes. `_metadata.*` queries now work against V2 file sources with the same
semantics as V1.
### How was this patch tested?
New `FileMetadataColumnsV2Suite` (24 tests) exercises read and projection paths
for Parquet/ORC/JSON/CSV/Text, forcing the V2 path via
`spark.sql.sources.useV1SourceList = ""`, and asserts the metadata struct values
against the underlying file's `java.io.File` stats.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56335][SQL] Implement SupportsMetadataColumns in FileTable [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #55320:
URL: https://github.com/apache/spark/pull/55320#issuecomment-5053096021
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56335][SQL] Implement SupportsMetadataColumns in FileTable [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #55320: [SPARK-56335][SQL] Implement SupportsMetadataColumns in FileTable
URL: https://github.com/apache/spark/pull/55320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56335][SQL] Implement SupportsMetadataColumns in FileTable [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #55320:
URL: https://github.com/apache/spark/pull/55320#issuecomment-4234462723
This is the 12th PR for [SPARK-56170](https://issues.apache.org/jira/browse/SPARK-56170)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org