You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zhang-arvin (via GitHub)" <gi...@apache.org> on 2026/09/22 03:40:19 UTC
[PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
zhang-arvin opened a new pull request, #10077:
URL: https://github.com/apache/paimon/pull/10077
### What is the purpose of the change
Fixes #9993. The automatic schema merge path bypasses the invariant that the explicit schema change path enforces: key columns must not change type.
### Brief change log
- `SchemaMergingUtils#mergeSchemas(TableSchema, RowType, ...)` merges every incoming field type and then re-attaches the **existing** `primaryKeys()` / `partitionKeys()` name lists, without ever validating that those columns kept their type. `SchemaManagerUtils` already rejects this for explicit changes via `assertNotUpdatingPartitionKeys` / `assertNotUpdatingPrimaryKeys`, but `FileSystemSchemaManager#mergeSchema` (i.e. every write-time automatic schema merge) called `mergeSchemas` directly and skipped those guards.
- Add `assertNotUpdatingKeyColumn`, invoked right before the merged `TableSchema` is constructed, which rejects the merge with an `UnsupportedOperationException` when a primary-key or partition column's type changed.
- Non-key column widening behaviour is unchanged.
### Why it matters
Widening a key column changes its value encoding, so the same logical key hashes into a different bucket. Concretely, for a bucketed primary-key table, `INT(-100000)` hashes to bucket 3 while `BIGINT(-100000)` hashes to bucket 13 — the same logical key ends up in two different buckets, so intra-bucket deduplication no longer applies and duplicate rows accumulate.
### Tests
- `testRejectTypeWideningOnPrimaryKeyColumn` — covers the reported `INT -> BIGINT` primary-key scenario, asserting the merge is rejected.
- `testRejectTypeWideningOnPartitionColumn` — same for a partition column.
- `testAllowTypeWideningOnNonKeyColumn` — guards against over-rejection: non-key widening still merges.
All 17 tests in `SchemaMergingUtilsTest` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #10077:
URL: https://github.com/apache/paimon/pull/10077#issuecomment-5771384947
The `paimon-core` failures in CI (`MockRESTCatalogTest.testDataTokenExpired`, `FormatTableCommitTest.testAbortFailureDoesNotReplaceInterruptedCleanupFailure`) are **pre-existing on upstream `master`**, not caused by this change.
Evidence — the same jobs fail on a clean `master` run:
- run https://github.com/apache/paimon/actions/runs/35684053322 (sha `0afddbd37`, branch `master`, no PR changes)
- its `Java / Core and integrations / JDK 8` job reports:
`MockRESTCatalogTest>RESTCatalogTest.testDataTokenExpired:2544 expected: <token is expired> but was: <Requested presigned URL validity exceeds the remaining REST credential lifetime after refresh.>`
This PR only touches `SchemaMergingUtils` (automatic merge-schema validation) and its test; neither failing class is on that path. `SchemaMergingUtilsTest` passes 17/17 locally, including the INT→BIGINT primary-key case from the report.
This comment was generated by an AI agent (Hermes Agent).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #10077:
URL: https://github.com/apache/paimon/pull/10077#discussion_r4068922688
##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaMergingUtils.java:
##########
@@ -68,6 +68,17 @@ public static TableSchema mergeSchemas(
return currentTableSchema;
}
+ // The automatic merge path must not relax the invariants enforced by the explicit schema
+ // change path (see SchemaManagerUtils#assertNotUpdatingPartitionKeys /
+ // #assertNotUpdatingPrimaryKeys). Widening a key column changes its value encoding, which
+ // silently rehashes it into another bucket, so the same logical key would land in two
+ // different buckets and bypass deduplication. Reject such merges instead of committing
+ // them.
+ assertNotUpdatingKeyColumn(
+ currentTableSchema, currentType, newRowType, "update", "partition column");
+ assertNotUpdatingKeyColumn(
+ currentTableSchema, currentType, newRowType, "update", "primary key");
Review Comment:
[P2] Update the Spark evolution tests for the new key-type contract
This correctly rejects a scenario that the existing Spark success tests still require: `paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:242` creates primary key `a INT`, feeds `MemoryStream[(Long, Date, Int)]` with type widening enabled, and expects the key to become BIGINT. The Spark 3.4 copy at `paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala:239` does the same. Current [Spark3 CI](https://github.com/apache/paimon/actions/runs/35684011729/job/106606936417) fails that test with `Cannot update primary key type from INT NOT NULL to BIGINT NOT NULL: [a]`, with the stack pointing to this new guard. Please update both successful evolution tests to retain the key type while evolving non-key columns, and add a separate streaming rejection case checking that schema metadata and existing rows remain unchanged. The guard should remain; the integration tests need to reflect the intended invariant in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "Akash3121 (via GitHub)" <gi...@apache.org>.
Akash3121 commented on code in PR #10077:
URL: https://github.com/apache/paimon/pull/10077#discussion_r4077885494
##########
paimon-core/src/test/java/org/apache/paimon/schema/SchemaMergingUtilsTest.java:
##########
@@ -144,6 +144,80 @@ public void testMergeSchemasWithoutTypeWidening() {
assertThat(fields.get(2).type()).isEqualTo(new IntType());
}
+ @Test
+ public void testRejectTypeWideningOnPrimaryKeyColumn() {
+ // An INT primary key must not be widened to BIGINT by the automatic merge path: INT
+ // values hash into different buckets than their BIGINT counterparts, so the same logical
+ // key would end up in two buckets and bypass intra-bucket deduplication.
+ DataField a = new DataField(0, "a", new IntType());
+ DataField b = new DataField(1, "b", new VarCharType(VarCharType.MAX_LENGTH));
+ TableSchema current =
+ new TableSchema(
+ 0,
+ Lists.newArrayList(a, b),
+ 1,
+ new ArrayList<>(),
+ Lists.newArrayList("a"),
+ new HashMap<>(),
+ "");
+
+ DataField aWidened = new DataField(-1, "a", new BigIntType());
+ RowType t = new RowType(Lists.newArrayList(aWidened, b));
+
+ assertThatThrownBy(() -> SchemaMergingUtils.mergeSchemas(current, t, true, false, true))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Cannot update primary key type");
+ }
+
+ @Test
+ public void testRejectTypeWideningOnPartitionColumn() {
+ // Partition columns are equally encoded into the partition path; widening them would
+ // silently create a second partition layout for the same logical key.
+ DataField a = new DataField(0, "a", new IntType());
+ DataField b = new DataField(1, "b", new VarCharType(VarCharType.MAX_LENGTH));
+ TableSchema current =
+ new TableSchema(
+ 0,
+ Lists.newArrayList(a, b),
+ 1,
+ Lists.newArrayList("b"),
+ new ArrayList<>(),
+ new HashMap<>(),
+ "");
+
+ DataField bWidened = new DataField(-1, "b", new VarCharType(100));
Review Comment:
This test is named `testRejectTypeWideningOnPartitionColumn` , but it changes `VARCHAR(MAX)` to `VARCHAR(100)` , which is a narrowing conversion, and it only succeeds as a merge candidate because `allowExplicitCast` is enabled below. Could we instead use INT -> BIGINT for the partition column and call `mergeSchemas` with `allowExplicitCast=false` ? That would directly verify that the normal type-widening path rejects partition-column type changes, independently of explicit-cast behavior.
##########
paimon-core/src/test/java/org/apache/paimon/schema/SchemaMergingUtilsTest.java:
##########
@@ -144,6 +144,80 @@ public void testMergeSchemasWithoutTypeWidening() {
assertThat(fields.get(2).type()).isEqualTo(new IntType());
}
+ @Test
+ public void testRejectTypeWideningOnPrimaryKeyColumn() {
+ // An INT primary key must not be widened to BIGINT by the automatic merge path: INT
+ // values hash into different buckets than their BIGINT counterparts, so the same logical
+ // key would end up in two buckets and bypass intra-bucket deduplication.
+ DataField a = new DataField(0, "a", new IntType());
+ DataField b = new DataField(1, "b", new VarCharType(VarCharType.MAX_LENGTH));
+ TableSchema current =
+ new TableSchema(
+ 0,
+ Lists.newArrayList(a, b),
+ 1,
+ new ArrayList<>(),
+ Lists.newArrayList("a"),
+ new HashMap<>(),
+ "");
+
+ DataField aWidened = new DataField(-1, "a", new BigIntType());
+ RowType t = new RowType(Lists.newArrayList(aWidened, b));
+
+ assertThatThrownBy(() -> SchemaMergingUtils.mergeSchemas(current, t, true, false, true))
+ .isInstanceOf(UnsupportedOperationException.class)
+ .hasMessageContaining("Cannot update primary key type");
+ }
+
+ @Test
+ public void testRejectTypeWideningOnPartitionColumn() {
+ // Partition columns are equally encoded into the partition path; widening them would
+ // silently create a second partition layout for the same logical key.
+ DataField a = new DataField(0, "a", new IntType());
+ DataField b = new DataField(1, "b", new VarCharType(VarCharType.MAX_LENGTH));
+ TableSchema current =
+ new TableSchema(
+ 0,
+ Lists.newArrayList(a, b),
+ 1,
+ Lists.newArrayList("b"),
+ new ArrayList<>(),
+ new HashMap<>(),
+ "");
+
+ DataField bWidened = new DataField(-1, "b", new VarCharType(100));
Review Comment:
The corresponding corrected test setup would look like:
```java
DataField b = new DataField(1, "b", new IntType());
// ...
DataField bWidened = new DataField(-1, "b", new BigIntType());
RowType t = new RowType(Lists.newArrayList(a, bWidened));
assertThatThrownBy(() -> SchemaMergingUtils.mergeSchemas(current, t, true, false, true))
.isInstanceOf(UnsupportedOperationException.class)
.hasMessageContaining("Cannot update partition column type");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on PR #10077:
URL: https://github.com/apache/paimon/pull/10077#issuecomment-6051498395
Addressed the [P2 Spark-fixture finding](https://github.com/apache/paimon/pull/10077#discussion_r4068901234) and @Akash3121's partition-test comment on head `fd69fb56a`:
**P2 — Spark evolution fixtures now follow the new key contract**
- `DataFrameWriteTestBase` Case 2 (both the plain and the `allowExplicitCast = true` variants) no longer widens the primary key `a`; it stays INT and only the non-key columns evolve (`c`: Long→Decimal, `b`: String→Date, plus `c: Long→Int`, `d: Map→String` via explicit cast). Expected rows updated accordingly.
- Added a rejection case in the same test: widening the primary key (`a` INT→BIGINT) now asserts `Cannot update primary key type`, and that the merge left the schema id and all previously committed rows unchanged.
- Updated the independent copy at `paimon-spark/paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala`: it now uses `MemoryStream[(Int, Date, Int)]` (mirrors the shared UT fix) instead of feeding a `Long` key into an INT primary-key table.
**Akash3121 — partition test**
- `testRejectTypeWideningOnPartitionColumn` now widens `b` from INT to BIGINT with `allowExplicitCast = false`, so it verifies the normal type-widening path rejects partition-column type changes independently of explicit-cast behavior (replacing the previous VARCHAR(MAX)→VARCHAR(100) narrowing).
`SchemaMergingUtilsTest` (including the three guard tests) passes locally; core compiles, checkstyle and spotless are green. The Spark module cannot be built in this checkout (its base predates a `paimon-core` API change in `paimon-spark-common`), so the Spark suites are left to CI.
This comment was generated by an AI agent (Hermes Agent).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10077:
URL: https://github.com/apache/paimon/pull/10077#issuecomment-6139126262
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10077:
URL: https://github.com/apache/paimon/pull/10077#issuecomment-5953678654
Reviewed current head `e3ae10af8792d200b5e63865e83de71e96f3dce9`. The key-type guard addresses a real correctness issue and belongs before schema persistence.
**[P2] Update the remaining Spark schema-evolution fixtures to the new key contract.** `DataFrameWriteTestBase` still widens primary key `a` from INT to BIGINT and expects successful writes, both in `Schema evolution: write data into Paimon` (Case 2) and its `allowExplicitCast = true` variant. I ran these two existing cases against this exact head: both fail with `Cannot update primary key type from INT NOT NULL to BIGINT NOT NULL: [a]`. This matches the current Spark CI failures; they are directly related to this change. Please keep the key type unchanged in the positive evolution cases and add explicit rejection/data-preservation assertions for key widening.
The shared UT `PaimonSinkTest` update now passes locally, but the independent copy in `paimon-spark-3.4/src/test/scala/org/apache/paimon/spark/PaimonSinkTest.scala` still uses `MemoryStream[(Long, Date, Int)]` against an INT primary key. Please update that copy too; it still fails in the Spark3 CI matrix.
Validation: 77 Core tests passed on JDK 8 with normal Maven checks. 55 focused tests passed on Spark 3.5.8, including the corrected streaming sink and V1/V2 merge suites. An independent actual path-write probe confirmed that rejected primary-key and partition widening leaves the schema ID, snapshot ID, and committed rows unchanged, while non-key widening subsequently commits and reads correctly. The same 55 tests passed on Spark 4.1.2 with normal Maven checks.
I found no additional production-logic regression in the guard after checking case-insensitive matching, retained nullability/names, and both path and catalog schema-persistence routes. Keep this PR open and finish the integration-test updates before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reject merge-schema type widening for primary-key and partition columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10077:
URL: https://github.com/apache/paimon/pull/10077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org