[PR] [SPARK-60341][SQL] Support internal metadata columns in changelog reads [spark]

["SanJSp (via GitHub)" <gi...@apache.org>]

SanJSp opened a new pull request, #59596:
URL: https://github.com/apache/spark/pull/59596

   ### What changes were proposed in this pull request?
   
   Add a default `Changelog.metadataColumns()` method and expose those columns through the internal
   `ChangelogTable` wrapper using `SupportsMetadataColumns`. Keep metadata resolution lazy and exclude
   metadata from `SELECT *`. Mark the CDC post-processing subtree and reject explicit user access to
   changelog metadata outside that subtree with `UNSUPPORTED_FEATURE.CHANGELOG_METADATA_COLUMN`.
   
   ### Why are the changes needed?
   
   Addresses [SPARK-60341](https://issues.apache.org/jira/browse/SPARK-60341).
   
   Connectors may provide row identity and row version as scan-generated metadata rather than data
   columns. Spark needs to resolve those references for carryover removal, update detection, and net
   change computation without adding them to the user-visible changelog schema. Direct user access is
   not supported by this change.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. CDC connectors can expose metadata needed for Spark's internal post-processing. Existing
   connectors retain the default empty metadata list. Changelog metadata remains absent from
   `SELECT *`, and explicit access through SQL or the Dataset API is rejected.
   
   ### How was this patch tested?
   
   Local verification passed on current Spark master using Java 17:
   
   - `ChangelogTableSuite`: 20 tests passed.
   - `ChangelogResolutionSuite`: 50 tests passed, including internal metadata resolution, `SELECT *`,
     SQL/Dataset metadata rejection, subqueries, and combined carryover removal and net changes.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Codex (GPT-5).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org