You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/09/07 12:53:00 UTC
[jira] [Assigned] (SPARK-59163) Reuse projected table state options across table loading and cache lookup
[ https://issues.apache.org/jira/browse/SPARK-59163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan reassigned SPARK-59163:
-----------------------------------
Assignee: Yan Yan
> Reuse projected table state options across table loading and cache lookup
> -------------------------------------------------------------------------
>
> Key: SPARK-59163
> URL: https://issues.apache.org/jira/browse/SPARK-59163
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Yan Yan
> Assignee: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> h3. Background
> SPARK-58392 and PR #57585 introduced table-state option projection for relation loading. During relation resolution, Spark already materializes the projected state-option map for the per-query table cache key.
> h3. Problem
> Some subsequent paths derive the same projection again before catalog loading or shared relation cache lookup. This repeats normalization and materialization and makes correctness depend on independently computed maps remaining equivalent.
> Safely reusing one projected map also requires it to remain stable after it is passed to connector code. {{CaseInsensitiveStringMap}} rejects direct mutation, but its normalized collection views were mutable. This makes an instance unsafe to retain as cache or deduplication identity after exposing it to a connector.
> h3. Proposed change
> Reuse the already-projected state-option map for the table cache key, catalog load, shared relation cache lookup, and execution refresh. Also apply a minor internal refactoring so method names make it explicit when callers pass a complete option map versus already-projected state options.
> Make {{CaseInsensitiveStringMap}} consistently read-only by backing its normalized entries with an unmodifiable map. The same projected instance can then be passed to catalog code and retained for cache or refresh-deduplication identity. Avoiding a reconstructed defensive copy also preserves original-case entries and the effective value already selected when input contains keys that differ only by case.
> h3. User-facing impact
> {{CaseInsensitiveStringMap}} is an {{@Experimental}} connector-facing API that was already intended to be read-only. Attempts to mutate it through {{keySet()}}, {{values()}}, or {{entrySet()}} now throw {{UnsupportedOperationException}}, consistent with its existing direct mutators. Read behavior is unchanged.
> Related issue: SPARK-58392
> PR: https://github.com/apache/spark/pull/58462
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org