[jira] [Created] (SPARK-59841) Align table and changelog state-option handling and metadata reuse

["Gengliang Wang (Jira)" <ji...@apache.org>]

Gengliang Wang created SPARK-59841:
--------------------------------------

             Summary: Align table and changelog state-option handling and metadata reuse
                 Key: SPARK-59841
                 URL: https://issues.apache.org/jira/browse/SPARK-59841
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


TableCatalog currently has different option-handling contracts for ordinary table reads and changelog reads:

* {{tableStateOptionKeys()}} declares the connector options that select table state. Spark passes only those options to {{loadTable(Identifier, TableContext, stateOptions)}} and uses them for table metadata reuse. Complete per-read options remain available for scan planning.
* {{loadChangelog(Identifier, ChangelogContext, options)}} receives all query options, and changelog resolution bypasses the corresponding state-aware table-loading/cache path. There is no separate changelog state-option declaration.

State options do reach CDC connectors, but Spark does not apply the same option projection and metadata-reuse rules to the two paths. This complicates deduplicating loads and refreshes, including the ongoing work on concurrent table refresh.

For example, a current-table read and a changes-from-version-1 read with matching table-state options should be able to share the underlying captured table state. Their scan options and changelog range still describe distinct reads. Different state options must remain isolated, and different changelog contexts must not be treated as equivalent reads.

The goal is to define consistent state-option, caching, and refresh semantics for ordinary and changelog reads, while retaining complete scan options and preserving the requested CDC context through refresh.

The API approach is undecided. This ticket does not prescribe changing the Changelog hierarchy, adding a baseTable accessor, or any particular wrapper or refresh-request abstraction. The implementation should be evaluated for compatibility with existing connector APIs for the Spark 4.3.0 release.

References:
* [TableCatalog option contracts|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java#L104]
* [Current changelog resolution|https://github.com/apache/spark/blob/29acd89ec5f4806c1f6a335d8d3136baf111cd08/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala#L382]
* [Exploratory prototype for discussion; API not decided|https://github.com/gengliangwang/spark/pull/20]

Target release: Spark 4.3.0.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org