You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/09/01 07:38:53 UTC
[PR] [common] Reject non-struct inner fields in variant shredding schema [paimon]
LuciferYang opened a new pull request, #9518:
URL: https://github.com/apache/paimon/pull/9518
### Purpose
close #9517
In a variant shredding schema, every field of an object's `typed_value` is itself a group of `value` / `typed_value`. `buildVariantSchema` cast each inner field to `RowType` without checking it, so a physical schema whose inner field is a scalar died on a bare `ClassCastException` naming two class names, instead of the `Invalid variant shredding schema: <rowType>` error the method raises for every other malformed shape. That path now checks the type and raises the same error as the rest.
Two checks in the same branch are removed because they cannot fire:
- `if (!(dataType instanceof RowType))` sits inside `case ROW:` of a switch on `field.type().getTypeRoot()`, and `RowType` is the only type in paimon-api with `DataTypeRoot.ROW`.
- `if (fields.isEmpty() || fields.stream().distinct().count() != fields.size())` tests the *outer* field list, not the inner struct it appears to guard. Emptiness is already rejected at the top of the method; duplicates cannot exist because `RowType`'s constructor calls `validateFields`, which throws on duplicate field names. Duplicate `typed_value` / `value` / `metadata` are caught by the `typedIdx != -1` style checks, and any other outer name falls into the throwing `default:` of the name switch.
An empty inner struct stays legal, exactly as before: the removed check never guarded the inner list, so nothing changes there. It yields a zero-length `objectSchema`, which is what `ShreddingUtils` and `VariantShreddingWriter` already consume by looping on `.length`.
### Tests
Both cases live in `PaimonShreddingUtilsTest`.
- `testBuildVariantSchemaRejectsNonStructInnerField` builds a schema that is valid except for a scalar inner field and asserts the message is the invalid-schema one. Verified red before the change, where it fails with `ClassCastException: org.apache.paimon.types.IntType cannot be cast to org.apache.paimon.types.RowType`. The message is what separates the two outcomes, since `ClassCastException` is itself a `RuntimeException`.
- `testBuildVariantSchemaAcceptsEmptyInnerStruct` pins that an empty `typed_value` struct produces a zero-length `objectSchema` rather than null, which is the shape the two consumers above depend on. This one passes before the change as well; it is there to hold that behavior still.
`mvn -pl paimon-common test` on JDK 8: 12467 tests, 0 failures, 0 errors. checkstyle, spotless, enforcer and rat run clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Reject non-struct inner fields in variant shredding schema [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9518:
URL: https://github.com/apache/paimon/pull/9518#issuecomment-5505571561
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Reject non-struct inner fields in variant shredding schema [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #9518:
URL: https://github.com/apache/paimon/pull/9518#issuecomment-5495352510
The `UTCase and ITCase Others on JDK 11` job was killed at the 6 hour default timeout, and it is not this change.
I pulled the job log: of the 826 test classes it started, exactly one never reported a result, `org.apache.paimon.operation.FileStoreCommitTest` in paimon-core, which began at 08:14 and was still running when the job was cancelled at 14:01. That class has no relation to variant shredding (it never touches `PaimonShreddingUtils` or a variant column), and it has a structural way to hang: it starts several `TestCommitThread`s and joins them with a bare `thread.join()`, while those threads retry inside `while (true)` loops with `FailingFileIO` failures injected, so a retry that cannot make progress spins with nothing to preempt it. The class is also randomized in thread count and data volume, and this job runs with a random timezone.
For the part that is mine, I re-ran every variant and shredding test locally on JDK 11 (Zulu 11.0.30), `paimon-common` plus `paimon-format`: 51 tests, 0 failures. `mvn -pl paimon-common clean test` on JDK 8 is also green at 12467 tests.
A re-run should clear the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Reject non-struct inner fields in variant shredding schema [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #9518:
URL: https://github.com/apache/paimon/pull/9518#issuecomment-5505592913
Thank you @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common] Reject non-struct inner fields in variant shredding schema [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9518:
URL: https://github.com/apache/paimon/pull/9518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org