You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "szehon-ho (via GitHub)" <gi...@apache.org> on 2026/10/20 22:15:37 UTC
[PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
szehon-ho opened a new pull request, #59660:
URL: https://github.com/apache/spark/pull/59660
### What changes were proposed in this pull request?
JIRA: [SPARK-60413](https://issues.apache.org/jira/browse/SPARK-60413)
Log a warning before `ResolveSchemaEvolution` applies pending schema changes to a DSv2 table reporting both `AUTOMATIC_SCHEMA_EVOLUTION` and `ACCEPT_ANY_SCHEMA`.
The warning is emitted once for the batch of schema changes. Writes without pending evolution and tables reporting only one capability remain quiet.
### Why are the changes needed?
`AUTOMATIC_SCHEMA_EVOLUTION` applies requested schema changes during write analysis, while `ACCEPT_ANY_SCHEMA` skips native write schema resolution and alignment. A write can therefore fail after updating the table schema. A nontransactional catalog retains those changes even when the write fails.
The interaction also arose in [Iceberg PR #17957](https://github.com/apache/iceberg/pull/17957). The warning identifies the affected data source and table and suggests disabling `ACCEPT_ANY_SCHEMA`.
### Does this PR introduce _any_ user-facing change?
Yes. When applying schema evolution to a table reporting both capabilities, Spark logs:
```text
Applying schema evolution to <data source> table <table>, which reports both
AUTOMATIC_SCHEMA_EVOLUTION and ACCEPT_ANY_SCHEMA capabilities. While
AUTOMATIC_SCHEMA_EVOLUTION natively resolves and evolves the schema on a write
schema evolution request, ACCEPT_ANY_SCHEMA skips native write schema resolution
and alignment. This may cause the write to fail after the table schema has been
updated. Consider disabling ACCEPT_ANY_SCHEMA.
```
### How was this patch tested?
Added an end-to-end test in `MergeIntoDataFrameSuite` using a nontransactional row-level catalog. It covers INSERT and MERGE, all four capability combinations, and schema evolution enabled and disabled: 16 configurations and 64 public write calls.
The test checks exact warning messages and counts, including multiple column additions in one write, later evolution requests, and repeated writes needing no further changes. It also verifies persisted rows and schemas, including schema changes retained after failed MERGE.
The focused test passed:
```sh
SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly org.apache.spark.sql.connector.MergeIntoDataFrameSuite -- -z "schema evolution warns"'
```
Whitespace, ASCII, and line-length checks also passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #59660:
URL: https://github.com/apache/spark/pull/59660#issuecomment-6303364800
@johanl-db @cloud-fan can you help take a look? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #59660:
URL: https://github.com/apache/spark/pull/59660#issuecomment-6372142716
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/0aa4f9136bccc40856a5af688e3d16f8b38ebfc4
- merged into branch-4.x https://github.com/apache/spark/commit/0f10db1231dea58eaedb29a6a0cdaaaaf4ea748c
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #59660:
URL: https://github.com/apache/spark/pull/59660#issuecomment-6352682909
Thanks for the review. Both inline nits are addressed in c3437c0aa2d. Whitespace, ASCII, and line-length checks passed; I have not rerun compilation or tests for this helper relocation and comment-only cleanup.
On the capability interaction, the intent is to warn about the risk while preserving existing connector behavior. This patch does not declare the combination invalid: a connector may provide its own resolution, and the INSERT case demonstrates that a write can succeed. Rejecting the combination or skipping Spark-side evolution would change that behavior. The warning therefore says the write may fail, rather than that it must fail.
The warning is emitted once per write that has pending schema changes, rather than once per table, because each such write can change catalog state before a later analysis failure. Writes that need no schema changes remain quiet. Disabling ACCEPT_ANY_SCHEMA is a suggestion for connectors that can use native resolution; connectors that require custom resolution need to account for this interaction. A TableCapability Javadoc note would help make the interaction discoverable, but this update leaves the warning policy and capability contract unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #59660:
URL: https://github.com/apache/spark/pull/59660#discussion_r4354181975
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala:
##########
@@ -1551,4 +1631,115 @@ class MergeIntoDataFrameSuite extends RowLevelOperationSuiteBase {
}
}
}
+
+ // Test helpers.
+
+ private def targetTableCol(colName: String): Column = {
Review Comment:
Fixed in c3437c0aa2d. Restored targetTableCol immediately after import testImplicits._; the bottom helper section now contains only the new helpers.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala:
##########
@@ -1188,6 +1196,78 @@ class MergeIntoDataFrameSuite extends RowLevelOperationSuiteBase {
}
}
+ test("schema evolution warns once per write only when both capabilities are present") {
+ val tableName = "non_txn.ns1.test_table"
+ for {
+ command <- Seq("INSERT", "MERGE")
+ (description, capabilities) <- Seq(
+ "neither capability" -> Set.empty[TableCapability],
+ "only ACCEPT_ANY_SCHEMA" -> Set(ACCEPT_ANY_SCHEMA),
+ "only AUTOMATIC_SCHEMA_EVOLUTION" -> Set(AUTOMATIC_SCHEMA_EVOLUTION),
+ "both capabilities" -> Set(AUTOMATIC_SCHEMA_EVOLUTION, ACCEPT_ANY_SCHEMA))
+ evolutionEnabled <- Seq(false, true)
+ } {
+ withClue(s"$command with $description, schema evolution=$evolutionEnabled: ") {
+ withTable(tableName) {
+ val acceptAnySchema = capabilities.contains(ACCEPT_ANY_SCHEMA)
+ val autoSchemaEvolution = capabilities.contains(AUTOMATIC_SCHEMA_EVOLUTION)
+ val warnOnEvolution = evolutionEnabled && acceptAnySchema && autoSchemaEvolution
+ sql(
+ s"""CREATE TABLE $tableName (pk INT) USING foo
+ |TBLPROPERTIES (
+ | 'accept-any-schema' = '$acceptAnySchema',
+ | 'auto-schema-evolution' = '$autoSchemaEvolution')
+ |""".stripMargin)
+ val initial = spark.createDataFrame(
+ Seq(Row(0)).asJava,
+ new StructType().add("pk", IntegerType))
+ val first = spark.createDataFrame(
+ Seq(Row(1, "a", true)).asJava,
+ initial.schema.add("data", StringType).add("active", BooleanType))
+ val next = spark.createDataFrame(
+ Seq(Row(2, "b", false, 2L)).asJava,
+ first.schema.add("score", LongType))
+
+ /** A write matching the table schema does not warn. */
Review Comment:
Fixed in c3437c0aa2d. Converted all four step comments to // line comments, including collapsing the repeating-write comment to one line. The test code and assertions are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #59660:
URL: https://github.com/apache/spark/pull/59660#issuecomment-6372150251
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #59660:
URL: https://github.com/apache/spark/pull/59660#discussion_r4343701462
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala:
##########
@@ -1551,4 +1631,115 @@ class MergeIntoDataFrameSuite extends RowLevelOperationSuiteBase {
}
}
}
+
+ // Test helpers.
+
+ private def targetTableCol(colName: String): Column = {
Review Comment:
**Nit (P3):** This helper used to sit right after `import testImplicits._` at the top of the class, and the new test doesn't call it. Could we leave it where it was and put only the new helpers in this section? Moving it adds unrelated churn to the diff and to blame, and the project guidance asks us not to reorganize existing members unless the change requires it.
**Verification:**
- **Inspection:** Compared with the merge base, the suite's diff contains no removal or re-addition of the targetTableCol helper; only the new test, its helpers, the test catalog, the catalog registration, and the import changes remain.
- **Compile:** The suite still compiles, with the existing MERGE tests resolving targetTableCol and the new warning test unchanged.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala:
##########
@@ -1188,6 +1196,78 @@ class MergeIntoDataFrameSuite extends RowLevelOperationSuiteBase {
}
}
+ test("schema evolution warns once per write only when both capabilities are present") {
+ val tableName = "non_txn.ns1.test_table"
+ for {
+ command <- Seq("INSERT", "MERGE")
+ (description, capabilities) <- Seq(
+ "neither capability" -> Set.empty[TableCapability],
+ "only ACCEPT_ANY_SCHEMA" -> Set(ACCEPT_ANY_SCHEMA),
+ "only AUTOMATIC_SCHEMA_EVOLUTION" -> Set(AUTOMATIC_SCHEMA_EVOLUTION),
+ "both capabilities" -> Set(AUTOMATIC_SCHEMA_EVOLUTION, ACCEPT_ANY_SCHEMA))
+ evolutionEnabled <- Seq(false, true)
+ } {
+ withClue(s"$command with $description, schema evolution=$evolutionEnabled: ") {
+ withTable(tableName) {
+ val acceptAnySchema = capabilities.contains(ACCEPT_ANY_SCHEMA)
+ val autoSchemaEvolution = capabilities.contains(AUTOMATIC_SCHEMA_EVOLUTION)
+ val warnOnEvolution = evolutionEnabled && acceptAnySchema && autoSchemaEvolution
+ sql(
+ s"""CREATE TABLE $tableName (pk INT) USING foo
+ |TBLPROPERTIES (
+ | 'accept-any-schema' = '$acceptAnySchema',
+ | 'auto-schema-evolution' = '$autoSchemaEvolution')
+ |""".stripMargin)
+ val initial = spark.createDataFrame(
+ Seq(Row(0)).asJava,
+ new StructType().add("pk", IntegerType))
+ val first = spark.createDataFrame(
+ Seq(Row(1, "a", true)).asJava,
+ initial.schema.add("data", StringType).add("active", BooleanType))
+ val next = spark.createDataFrame(
+ Seq(Row(2, "b", false, 2L)).asJava,
+ first.schema.add("score", LongType))
+
+ /** A write matching the table schema does not warn. */
Review Comment:
**Nit (P3):** These step comments (here and at 1236, 1241, and 1246-1248) use Scaladoc `/** ... */` delimiters on statements inside the test body. Scaladoc documents definitions, and the rest of the suite, including the new `// DataFrame columns supply ...` comment in `writeWithSchemaEvolution`, uses `//` for in-body comments. Could we switch these to `//` line comments? The three-line block for "Repeating the write does not warn." also fits on one line.
**Verification:**
- **Inspection:** Inside the new schema-evolution warning test, every step comment is a // line comment and no /** ... */ block precedes a statement, while the test's code and assertions are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #59660: [SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema
URL: https://github.com/apache/spark/pull/59660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org