You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/08/22 00:36:56 UTC
[PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
szehon-ho opened a new pull request, #58222:
URL: https://github.com/apache/spark/pull/58222
### What changes were proposed in this pull request?
Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of
`docs/sql-migration-guide.md`, documenting the behavior change made by
SPARK-58517 (#57722):
- Declarative Pipelines now honors `spark.sql.caseSensitive` when inferring and
evolving pipeline table schemas. Under case-insensitive resolution (the
default), column names differing only in case identify the same column, so
flows contribute a single column rather than one per spelling, and the note
states which spelling survives.
- All flows writing to one pipeline table must agree on the effective
`spark.sql.caseSensitive`, or the update fails with the new
`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` error condition.
Documentation only; no code change.
### Why are the changes needed?
SPARK-58517 first ships in 4.3.0 (merged to `master`, `branch-4.x`, and
`branch-4.3`), and the behavior it changed has been in place since 4.1.0:
`SchemaMergingUtils.mergeSchemas` merged positionally, leaving
`StructType.merge`'s `caseSensitive = true` default; `DataflowGraph.inferredSchema`
merged with `reduce(SchemaMergingUtils.mergeSchemas)`; and
`SchemaInferenceUtils.diffSchemas` keyed columns on exact field names. Users
upgrading from 4.1.x or 4.2.x will see two observable differences that a
migration note should cover:
- A target schema that previously evolved to carry both spellings, then failed
later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`,
now folds into a single column.
- A pipeline whose flows disagree on `spark.sql.caseSensitive` now fails up
front with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change. It documents a user-facing change that
was already made by SPARK-58517.
### How was this patch tested?
No tests; documentation only. The error condition names in the notes were
checked against the tree (`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` in
`error-conditions.json`, `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA` in
`GraphErrors.scala`), as was the "lowest flow identifier wins" rule
(`SchemaInferenceUtils.inferSchemaFromFlows` sorts by
`_.identifier.unquotedString`). The link form
`declarative-pipelines-programming-guide.html` matches the existing link in
`docs/index.md`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 (Cursor)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58222:
URL: https://github.com/apache/spark/pull/58222#issuecomment-5401407390
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/5a26f48078e4db37ea399aa68070c87af62528d5
- merged into branch-4.x https://github.com/apache/spark/commit/030b1e16a8790e12b229b1ad158665616abf4003
- merged into branch-4.3 https://github.com/apache/spark/commit/06ce0356f3f443c265b1d71a97736698c2aaed70
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #58222: [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity
URL: https://github.com/apache/spark/pull/58222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58222:
URL: https://github.com/apache/spark/pull/58222#issuecomment-5379400217
+1, thank you @szehon-ho and @anew!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
Posted by "AnishMahto (via GitHub)" <gi...@apache.org>.
AnishMahto commented on code in PR #58222:
URL: https://github.com/apache/spark/pull/58222#discussion_r3836794495
##########
docs/sql-migration-guide.md:
##########
@@ -48,6 +48,8 @@ license: |
- Since Spark 4.3, the Spark-recognized time-travel options (`versionAsOf` and `timestampAsOf`, or the keys configured by `spark.sql.timeTravelVersionKey` and `spark.sql.timeTravelTimestampKey`) are rejected with `UNSUPPORTED_FEATURE.TIME_TRAVEL` on catalog-backed Data Source V2 writes, including table creation and replacement through `DataFrameWriterV2`, because writes must target the current table state rather than a historical version. Previously, Spark passed them to the connector as ordinary write options.
- Since Spark 4.3, `HAVING` is evaluated before window functions when the `SELECT` list also contains generator functions such as `explode`. Previously, window functions could include groups removed by `HAVING` and produce incorrect results.
- Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
+- Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of being added alongside it. Previously such names were always treated as distinct, producing a table schema that Spark's own resolver could not disambiguate and that could fail later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`. When two flows' columns fold together but their types are incompatible, the update now fails at validation with `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA`. Where the spellings differ, the surviving one comes from the flow with the lowest identifier, or, whe
n the target table already has the column, its persisted spelling. Set `spark.sql.caseSensitive` to `true` to keep names differing only in case distinct, as before.
Review Comment:
Optional since maybe its unnecessarily specific:
> when the target table already has the column
Only applies to incremental target tables (STs) and not MVs, which re-infer schema every update. And if a table defined in the pipeline explicitly defines its schema, the spelling there is always respected over the inferred spelling from flows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58222:
URL: https://github.com/apache/spark/pull/58222#discussion_r3846450790
##########
docs/sql-migration-guide.md:
##########
@@ -48,6 +48,8 @@ license: |
- Since Spark 4.3, the Spark-recognized time-travel options (`versionAsOf` and `timestampAsOf`, or the keys configured by `spark.sql.timeTravelVersionKey` and `spark.sql.timeTravelTimestampKey`) are rejected with `UNSUPPORTED_FEATURE.TIME_TRAVEL` on catalog-backed Data Source V2 writes, including table creation and replacement through `DataFrameWriterV2`, because writes must target the current table state rather than a historical version. Previously, Spark passed them to the connector as ordinary write options.
- Since Spark 4.3, `HAVING` is evaluated before window functions when the `SELECT` list also contains generator functions such as `explode`. Previously, window functions could include groups removed by `HAVING` and produce incorrect results.
- Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
+- Since Spark 4.3, [Declarative Pipelines](declarative-pipelines-programming-guide.html) honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names that differ only in case now identify the same column: flows writing to one table contribute a single column rather than one per spelling, and a column that differs only in case from one already persisted in the target is written to that column instead of being added alongside it. Previously such names were always treated as distinct, producing a table schema that Spark's own resolver could not disambiguate and that could fail later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`. When two flows' columns fold together but their types are incompatible, the update now fails at validation with `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA`. Where the spellings differ, the surviving one comes from the flow with the lowest identifier, or, whe
n the target table already has the column, its persisted spelling. Set `spark.sql.caseSensitive` to `true` to keep names differing only in case distinct, as before.
Review Comment:
Thanks — updated. The persisted-spelling clause now applies only to incremental streaming tables; the note also covers explicit table schemas (declared spelling wins) and materialized views (schema is re-inferred each update).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org