You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2026/07/04 04:49:18 UTC
[PR] [spark] Fail fast on incompatible query schema in v2 write [paimon]
kerwin-zk opened a new pull request, #8456:
URL: https://github.com/apache/paimon/pull/8456
### Purpose
Since #6281, Paimon tables declare `ACCEPT_ANY_SCHEMA` for Spark v2 write, so Spark skips its own output resolution (`DataSourceV2Relation#skipSchemaResolution`) and the query is aligned to the table schema by `PaimonAnalysis` instead, which is only injected when `PaimonSparkSessionExtensions` is configured.
If a write is planned without the extensions (e.g. `spark.paimon.requiredSparkConfsCheck.enabled=false` is set, or the plan is constructed by a third-party engine), a type-mismatched INSERT reaches the writer as-is: the writer interprets the incoming `UnsafeRow` with the table's row type, so a slot holding a DOUBLE gets read as a string's (offset, size). Depending on the bits this throws `NegativeArraySizeException` deep inside the format writer, crashes the executor with an out-of-bounds `Unsafe` read, or **silently writes garbage strings that can be committed** (we hit all three in a production case where a query wrote `SUM(...)` doubles into STRING columns).
This PR adds a defense-in-depth fail-fast in `PaimonV2WriteBuilder#build`: the query schema must be positionally type-compatible with the table schema (reusing `DataType.equalsIgnoreCompatibleNullability`; CHAR/VARCHAR are normalized to STRING since they share the string binary layout). Mismatches now fail at planning time with a clear message pointing to `PaimonSparkSessionExtensions`.
The check is skipped for row-level operations (`copyOnWriteScan` is set) and for `write.merge-schema` writes, which intentionally diverge from the current table schema and are validated elsewhere.
### Tests
- New `PaimonV2WriteSchemaValidationTest`: without the extensions, a schema-matching INSERT still succeeds, while type-mismatched and column-count-mismatched INSERTs fail fast with the new error (previously the type mismatch failed at execution time with `NegativeArraySizeException`).
- Regression: `PaimonOptionTest`, `PaimonConfigCheckTest`, `V2WriteMergeSchemaTest`, `V2WriteRequireDistributionTest`, `PaimonDynamicPartitionOverwriteCommandTest`, `SparkWriteITCase` pass on Spark 3.5 (Scala 2.12); `PaimonV2WriteSchemaValidationTest` + `V2WriteMergeSchemaTest` pass on Spark 4.x (Scala 2.13).
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fail fast on incompatible query schema in v2 write [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3523236194
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2WriteBuilder.scala:
##########
@@ -55,4 +61,63 @@ class PaimonV2WriteBuilder(table: FileStoreTable, dataSchema: StructType, option
}
override def partitionRowType(): RowType = table.schema().logicalPartitionType()
+
+ /**
+ * Fail fast when the query schema is binary-incompatible with the table schema.
+ *
+ * Paimon tables declare `ACCEPT_ANY_SCHEMA`, so Spark skips its own output resolution and the
+ * query is aligned to the table schema by `PaimonAnalysis` instead, which is only injected when
+ * `PaimonSparkSessionExtensions` is configured. If a write is planned without it, a
+ * type-mismatched query would reach the writer as-is and be interpreted with the table's row
+ * type, silently corrupting data or failing with errors like `NegativeArraySizeException` deep
+ * inside the format writer.
+ */
+ private def validateDataSchema(): Unit = {
+ // Row-level operations write the table's own rows back; schema evolution intentionally
+ // diverges from the current table schema and is validated when merging.
+ if (copyOnWriteScan.nonEmpty || SchemaEvolutionHelper.mergeSchemaEnabled(options)) {
+ return
+ }
+
+ val expectedFields = SparkTypeUtils.fromPaimonRowType(table.rowType()).fields
+ val actualFields = SparkSystemColumns.filterSparkSystemColumns(dataSchema).fields
Review Comment:
This does not catch the PR's own type-mismatch case. On this branch, running
`mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest -Dtest=none test`
still fails at `PaimonOptionTest.scala:331`: the insert reaches `PaimonV2DataWriter` and aborts the Spark job with a `SparkException` caused by `NegativeArraySizeException`, instead of failing here with the new `PaimonSparkSessionExtensions` validation message. That means `dataSchema` is not a reliable proxy for the physical query rows in this write path (or the validation is happening at the wrong layer), so the original fail-fast/corruption path remains unfixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fail fast on incompatible query schema in v2 write [paimon]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3523356787
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/write/PaimonV2WriteBuilder.scala:
##########
@@ -55,4 +61,63 @@ class PaimonV2WriteBuilder(table: FileStoreTable, dataSchema: StructType, option
}
override def partitionRowType(): RowType = table.schema().logicalPartitionType()
+
+ /**
+ * Fail fast when the query schema is binary-incompatible with the table schema.
+ *
+ * Paimon tables declare `ACCEPT_ANY_SCHEMA`, so Spark skips its own output resolution and the
+ * query is aligned to the table schema by `PaimonAnalysis` instead, which is only injected when
+ * `PaimonSparkSessionExtensions` is configured. If a write is planned without it, a
+ * type-mismatched query would reach the writer as-is and be interpreted with the table's row
+ * type, silently corrupting data or failing with errors like `NegativeArraySizeException` deep
+ * inside the format writer.
+ */
+ private def validateDataSchema(): Unit = {
+ // Row-level operations write the table's own rows back; schema evolution intentionally
+ // diverges from the current table schema and is validated when merging.
+ if (copyOnWriteScan.nonEmpty || SchemaEvolutionHelper.mergeSchemaEnabled(options)) {
+ return
+ }
+
+ val expectedFields = SparkTypeUtils.fromPaimonRowType(table.rowType()).fields
+ val actualFields = SparkSystemColumns.filterSparkSystemColumns(dataSchema).fields
Review Comment:
Thanks for checking! I believe that run picked up a stale `paimon-spark-common_2.12-1.5-SNAPSHOT` from the local repository instead of the one from this branch, so the PR's change was never on the classpath.
`-Pfast-build` explicitly activates a profile, which disables the `activeByDefault` activation of the `spark3` profile (standard Maven behavior). Without `spark3`, the `paimon-spark-common` modules drop out of the reactor — the reactor summary of that exact command only contains:
```
[INFO] Paimon : Spark [pom]
[INFO] Paimon : Spark : UT : 2.12 [jar]
```
so `-am` never compiles this PR's change to `PaimonV2WriteBuilder`, and the test runs against whatever `1.5-SNAPSHOT` is installed in `~/.m2`. If that snapshot predates this PR, the test fails with exactly the signature you saw — I actually produced the same message on purpose while developing (ran the new test against a build with the fix stashed to verify it was red first):
```
Expected exception java.lang.RuntimeException to be thrown,
but org.apache.spark.SparkException was thrown (PaimonOptionTest.scala:331)
```
Adding the spark3 profile brings the common modules back into the reactor:
```
mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build,spark3 -DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest -Dtest=none clean test
```
and the suite passes (your exact command also passes once the branch's `paimon-spark-common` is installed to the local repo; the PR CI, which builds from scratch, is green on both 2.12 and 2.13 as well):
```
- Paimon V2 write: fail fast on incompatible query schema without Paimon extensions
Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
```
On the layering question: for this insert path `LogicalWriteInfo.schema()` is the analyzed query's output schema, which is exactly the row layout the writer receives, so validating it against the table row type in `build()` catches the mismatch on the driver before any task runs. Row-level operations (`copyOnWriteScan`) and `write.merge-schema` writes are exempted since their schemas intentionally diverge and are validated elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fail fast on incompatible query schema in v2 write [paimon]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk closed pull request #8456: [spark] Fail fast on incompatible query schema in v2 write
URL: https://github.com/apache/paimon/pull/8456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fail fast on incompatible query schema in v2 write [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8456:
URL: https://github.com/apache/paimon/pull/8456#discussion_r3527492282
##########
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/PaimonOptionTest.scala:
##########
@@ -305,3 +305,41 @@ class PaimonConfigCheckTest extends SparkFunSuite {
}
}
}
+
+class PaimonV2WriteSchemaValidationTest extends SparkFunSuite with SparkVersionSupport {
+
+ test("Paimon V2 write: fail fast on incompatible query schema without Paimon extensions") {
+ assume(gteqSpark3_4)
+ val spark = SparkSession
+ .builder()
+ .master("local[2]")
+ .config("spark.sql.catalog.paimon", "org.apache.paimon.spark.SparkCatalog")
+ .config("spark.sql.catalog.paimon.warehouse", Utils.createTempDir.getCanonicalPath)
+ .config("spark.paimon.requiredSparkConfsCheck.enabled", "false")
+ .config("spark.paimon.write.use-v2-write", "true")
+ .getOrCreate()
+ try {
+ spark.sql("USE paimon")
+ spark.sql("CREATE TABLE T (a STRING, b STRING, pt STRING) PARTITIONED BY (pt)")
+
+ // A query schema that matches the table schema still works without the extensions.
+ spark.sql("INSERT INTO T VALUES ('x', 'y', '20260601')")
+ assert(spark.sql("SELECT * FROM T").collect() === Array(Row("x", "y", "20260601")))
+
+ // Type mismatch: without PaimonAnalysis no cast is added, the write must fail fast
+ // instead of corrupting data or throwing NegativeArraySizeException in the writer.
+ val typeMismatch = intercept[RuntimeException] {
Review Comment:
Running the added suite locally with `mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PaimonV2WriteSchemaValidationTest -Dtest=none test` fails at this assertion: this insert still reaches the writer and throws a `SparkException` caused by `NegativeArraySizeException`, rather than the new fail-fast error mentioning `PaimonSparkSessionExtensions`. That means the validation is not protecting the dynamic-partition overwrite path covered by this test. Could you adjust the guard/test so this statement is actually rejected before execution reaches the data writer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org