You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andreas Neumann (Jira)" <ji...@apache.org> on 2026/07/25 02:01:00 UTC
[jira] [Created] (SPARK-58347) Thread conf.resolver through ColumnSelection.applyToSchema instead of a caseSensitive boolean
Andreas Neumann created SPARK-58347:
---------------------------------------
Summary: Thread conf.resolver through ColumnSelection.applyToSchema instead of a caseSensitive boolean
Key: SPARK-58347
URL: https://issues.apache.org/jira/browse/SPARK-58347
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
Follow-up from review of SPARK-58313 (PR #57490). AutoCDC column-name resolution is currently driven by a caseSensitive: Boolean that each call site derives from conf.caseSensitiveAnalysis, then re-derives a resolver (or uses StructType.getFieldIndex / getFieldIndexCaseInsensitive) internally. A reviewer suggested threading conf.resolver all the way through for futureproofing, so resolution semantics live in one place rather than being reconstructed from a boolean at each layer.
The blocker for doing this within SPARK-58313 was scope: ColumnSelection.applyToSchema (and its helper lookupFieldIndices) take a caseSensitive: Boolean and rely on StructType.getFieldIndex / getFieldIndexCaseInsensitive, which are boolean-driven. Threading a Resolver requires reworking applyToSchema/lookupFieldIndices to be resolver-based and updating all callers.
Scope:
- Rework ColumnSelection.applyToSchema and lookupFieldIndices to accept/use a Resolver (or a small resolution abstraction) instead of a Boolean.
- Update all callers to pass conf.resolver: Scd1BatchProcessor (2 sites), Scd2BatchProcessor (preprocess projection + computeTrackedHistoryColumns), and Flow (userSelectedSchema).
- Scd2BatchProcessor.computeTrackedHistoryColumns can then take a Resolver directly instead of deriving one from a caseSensitive boolean.
This is a pure refactor with no behavior change (case-sensitive and case-insensitive resolution must remain identical); it also touches the shared SCD1 code paths, which is why it is intentionally deferred out of SPARK-58313.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org