You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "PDGGK (via GitHub)" <gi...@apache.org> on 2026/09/01 13:02:33 UTC

[PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

PDGGK opened a new pull request, #9532:
URL: https://github.com/apache/paimon/pull/9532

   ### Purpose
   
   Adaptive variant shredding inference throws and fails the write when a node's evidence is gone but its selected schema is still a `ROW`.
   
   `finalizeAdaptiveSchema` substitutes the previously *selected* schema when the current file has no evidence for a node:
   
   ```java
   if (combined == null || combined instanceof VariantType) {
       if (current != null && !(current instanceof VariantType)) {
           combined = current;
       } else if (previousSelected != null) {
           combined = previousSelected;          // InferVariantShreddingSchema:576-577
       } else {
           return DataTypes.VARIANT();
       }
   }
   ```
   
   Execution then falls into the `RowType` branch, which reads that substitute as if it were evidence:
   
   ```java
   double ratio = rootValueCount == 0 ? 0 : getFieldCount(field) / rootValueCount;   // :592
   ```
   
   Selected schemas carry no per-field counts — `finalizeSimpleSchema` clears them, and the adaptive `RowType` branch rebuilds fields with the description-free three-argument `DataField` constructor — so `getFieldCount` throws:
   
   ```
   java.lang.IllegalStateException: Field 'x' is missing count in description. This should not happen during schema inference.
     at InferVariantShreddingSchema.getFieldCount(InferVariantShreddingSchema.java:394)
     at InferVariantShreddingSchema.finalizeAdaptiveSchema(InferVariantShreddingSchema.java:592)
     at InferVariantShreddingSchema.finalizeAdaptiveSchema(InferVariantShreddingSchema.java:615)
     at InferVariantShreddingSchema.inferAdaptive(InferVariantShreddingSchema.java:134)
     at VariantShreddingInferenceSession.inferSchema(VariantShreddingInferenceSession.java:74)
   ```
   
   Getting there needs three things in one rolling writer with `variant.inferShreddingSchema=true` and `variant.shredding.inferenceMode=adaptive`: a node drifts across a type family, which degrades its combined evidence to `VARIANT` while the schema selected for it is a `ROW`; and it is then absent from the next file, so there is no current evidence either. Two files cannot do it — when a node is absent, `combineEvidence` falls back to the previous file's *evidence*, which still carries counts — so it takes three.
   
   The exception comes out of `InferShreddingWritePlanWriter` uncaught, so the file fails to write at all.
   
   Three files of a `ROW<v VARIANT>` table, at the default options, are enough:
   
   | file | rows | result |
   |---|---|---|
   | 1 | `{"k":1,"p":5}` | plans, `p` selected as `BIGINT` |
   | 2 | `{"k":1,"p":{"x":1}}` | plans, `p` selected as `ROW<x BIGINT>`, combined evidence for `p` degrades to `VARIANT` |
   | 3 | `{"k":1}` | **throws** |
   
   A root-level variant of the same shape — `42`, then `{"a":1}`, then SQL `NULL` — fails the same way with `Field 'a'`.
   
   ### Summary and Changelog
   
   Return `previousSelected` instead of assigning it to `combined`. It is already a finalized selection: it has been through admission, retention and the field budget once, and with no evidence for this file there is nothing to re-threshold it against. Carrying it forward unchanged is also what the retention path does for a node whose evidence is still a `RowType`.
   
   ### Tests
   
   `InferVariantShreddingSchemaTest#testAdaptiveInferenceKeepsSelectedRowWhenEvidenceDegradedAndNodeIsAbsent` drives the three files above and asserts `p` keeps `ROW<x BIGINT>`. On `master` it errors with the `IllegalStateException` above, and the other 24 cases in the class are unaffected either way.
   
   Four negative controls, run against a build of this ref, confirm the trigger is that specific combination rather than absence alone — no drift and `p` merely disappears; drift but `p` still present in file 3; object all the way, which is the shape every existing adaptive test uses; and root scalar to scalar to NULL. All four pass before and after.
   
   The reason no existing test catches this: the four adaptive cases in `InferVariantShreddingSchemaTest` and `InferVariantShreddingWriteTest` all keep the root variant an object across every file, so combined evidence stays a `RowType` and line 577 is never reached with a `RowType` in `previousSelected`. `testAdaptiveInferenceWidensScalarSelectedFromPriorEvidence` comes closest — its `second` column has no current evidence in round 2 — but its combined evidence there is the previous file's evidence, which still carries counts.
   
   `mvn test -pl paimon-common -Dtest='org.apache.paimon.data.variant.**'` — 52 tests, and `-pl paimon-format -Dtest=InferVariantShreddingWriteTest` — 18 tests, all passing. `spotless:check` and `checkstyle:check` clean.
   
   ### Tests API and Compatibility
   
   No API, format or configuration change. Only the path that currently throws behaves differently; every schema that infers successfully today infers the same way.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9532:
URL: https://github.com/apache/paimon/pull/9532#discussion_r3911805771


##########
paimon-common/src/main/java/org/apache/paimon/data/variant/InferVariantShreddingSchema.java:
##########
@@ -654,6 +658,43 @@ private DataType finalizeAdaptiveSchema(
         return selectScalarType(combined, current, previousSelected);
     }
 
+    /**
+     * Carries a previously selected schema forward for a node the current file has no evidence for.
+     * The selection is already final, so nothing is re-thresholded, but its nodes still consume the
+     * shared width budget. The entry unit for this node has already been spent by the caller, so
+     * this mirrors what finalizeAdaptiveSchema does from that point on: a child is entered only
+     * while budget remains, entering it spends one unit, and a child that exhausts the budget
+     * becomes VARIANT while its field or array container is still kept.
+     */
+    private DataType retainSelectedSchema(DataType selected, MaxFields maxFields) {
+        if (selected instanceof RowType) {
+            List<DataField> fields = new ArrayList<>();
+            for (DataField field : ((RowType) selected).getFields()) {
+                if (maxFields.remaining <= 0) {
+                    break;
+                }
+                maxFields.remaining--;
+                DataType retained =
+                        maxFields.remaining <= 0
+                                ? DataTypes.VARIANT()
+                                : retainSelectedSchema(field.type(), maxFields);
+                fields.add(new DataField(fields.size(), field.name(), retained));
+            }
+            return fields.isEmpty() ? DataTypes.VARIANT() : new RowType(fields);
+        }
+        if (selected instanceof ArrayType) {
+            maxFields.remaining--;
+            DataType element =
+                    maxFields.remaining <= 0
+                            ? DataTypes.VARIANT()
+                            : retainSelectedSchema(
+                                    ((ArrayType) selected).getElementType(), maxFields);
+            return new ArrayType(element);
+        }
+        maxFields.remaining--;

Review Comment:
   [P2] Do not charge retained VARIANT leaves twice
   
   The caller already spends the node entry unit before `retainSelectedSchema` is reached. For a `VariantType`, there is no typed-value child to spend another unit on; the normal `finalizeAdaptiveSchema` path likewise returns `VARIANT` after only the entry debit. This branch therefore overcharges retained `VARIANT` leaves.
   
   A minimal boundary case reproduces the regression: use two top-level variant columns with `maxSchemaWidth = 3`, infer/commit `[null, 5]`, then infer `[null, 6]`. The first retained `VARIANT` should cost one unit and leave two for the second `BIGINT`, but this extra decrement exhausts the budget and the second column becomes untyped `VARIANT`. I confirmed that assertion fails against this head while the existing 27 tests pass. Please special-case `VariantType` here and add the shared-budget regression.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9532:
URL: https://github.com/apache/paimon/pull/9532#discussion_r3909659423


##########
paimon-common/src/main/java/org/apache/paimon/data/variant/InferVariantShreddingSchema.java:
##########
@@ -654,6 +658,37 @@ private DataType finalizeAdaptiveSchema(
         return selectScalarType(combined, current, previousSelected);
     }
 
+    /**
+     * Carries a previously selected schema forward for a node the current file has no evidence for.
+     * The selection is already final, so nothing is re-thresholded, but its nodes still consume the
+     * shared width budget and are dropped once it runs out, on the same terms as the
+     * evidence-driven walk above: one unit per node, VARIANT once the budget is gone.
+     */
+    private DataType retainSelectedSchema(DataType selected, MaxFields maxFields) {
+        if (selected instanceof RowType) {
+            List<DataField> fields = new ArrayList<>();
+            for (DataField field : ((RowType) selected).getFields()) {
+                maxFields.remaining--;
+                if (maxFields.remaining <= 0) {

Review Comment:
   [P2] Preserve the retained field/container at the last budget unit
   
   This does not match the evidence-driven walk described above. When one unit remains, that walk enters the child, decrements the unit to zero, returns `VARIANT`, and still adds the parent `DataField`; this helper decrements first and breaks before adding it. If it was the only field, line 678 then collapses the whole retained `RowType` to the root `VARIANT`. The Array branch similarly collapses the array instead of retaining `ARRAY<VARIANT>`.
   
   I reproduced this with `maxSchemaWidth = 7` and two Variant columns. Round 1 is `(1, 5)`, round 2 is `(1, {q:1})`, and round 3 is `({x:1,y:1}, null)`. Column `a` consumes five units, leaving exactly the root plus one field unit for retained column `b`. The updated code returns an untyped root for `b` and loses `q`; a focused assertion that `b` still contains `q` fails.
   
   Please mirror `finalizeAdaptiveSchema` at exhaustion: retain the row field/array container and downgrade the exhausted child to `VARIANT`, rather than breaking/returning before preserving the parent shape.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "PDGGK (via GitHub)" <gi...@apache.org>.
PDGGK commented on PR #9532:
URL: https://github.com/apache/paimon/pull/9532#issuecomment-5549954938

   Reproduced your boundary before changing anything: two variant columns at `maxSchemaWidth = 3`, `[null, 5]` then `[null, 6]`, and `b` came back an untyped root instead of `BIGINT`. Fixed in `ab8f8fb6f`.
   
   `retainSelectedSchema` now returns a `VariantType` without the second decrement, since the caller has already spent the entry unit and there is no typed child to spend another on.
   
   `testRetainedVariantLeafIsNotChargedTwice` asserts your case. Removing only the `VariantType` branch, keeping every other part of this PR, fails that one test and leaves the other 27 green.
   
   `mvn test -pl paimon-common -Dtest='org.apache.paimon.data.variant.**'` — 58 tests, all passing; `spotless:check` and `checkstyle:check` clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9532:
URL: https://github.com/apache/paimon/pull/9532


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9532:
URL: https://github.com/apache/paimon/pull/9532#issuecomment-5747792616

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "PDGGK (via GitHub)" <gi...@apache.org>.
PDGGK commented on PR #9532:
URL: https://github.com/apache/paimon/pull/9532#issuecomment-5502651730

   Both P2s confirmed and fixed in `20d2ec77` — I reproduced each of them against an unmodified build first.
   
   **Width budget.** With `maxSchemaWidth = 8`, `r` came out `ROW<value BYTES, typed_value BIGINT>`, so the second column did get to spend what the carried-forward `ROW<x BIGINT>` was holding. The branch now walks the retained schema instead of returning it.
   
   **Exhaustion semantics.** Your second point was right about my first attempt: it decremented before its guard, so the field that consumed the final unit was dropped rather than kept with a `VARIANT` child. With `maxSchemaWidth = 7` and your rounds, `b` came back as an untyped root and lost `q`. It now enters a child only while budget remains, spends the unit on entry, and downgrades an exhausted child to `VARIANT` with its field or array container intact — the same shape as the loop in `finalizeAdaptiveSchema`.
   
   Three controls, so each test is pinned to the thing it covers rather than to the crash:
   
   | source | what fails |
   |---|---|
   | `master` | both retention tests error with the `IllegalStateException` |
   | early return, no debiting | only `testRetainedSchemaStillConsumesTheSharedWidthBudget`, as an assertion failure |
   | decrement before the guard | only `testRetainedSchemaKeepsItsFieldsAtTheLastBudgetUnit` |
   
   `mvn test -pl paimon-common -Dtest='org.apache.paimon.data.variant.**'` — 54 tests, and `-pl paimon-format -Dtest=InferVariantShreddingWriteTest` — 18 tests, all passing. `spotless:check` and `checkstyle:check` clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Keep the selected variant schema when its evidence is gone [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9532:
URL: https://github.com/apache/paimon/pull/9532#discussion_r3909588215


##########
paimon-common/src/main/java/org/apache/paimon/data/variant/InferVariantShreddingSchema.java:
##########
@@ -574,7 +574,11 @@ private DataType finalizeAdaptiveSchema(
             if (current != null && !(current instanceof VariantType)) {
                 combined = current;
             } else if (previousSelected != null) {
-                combined = previousSelected;
+                // This node has no evidence in the current file. A previously selected schema is
+                // not
+                // evidence - its fields carry no counts - so it cannot be run through admission and
+                // retention again; carry it forward unchanged.
+                return previousSelected;

Review Comment:
   [P2] Count preserved descendants against the shared width budget
   
   `maxFields` is shared across every Variant path in this inference, but this early return only consumes the single unit decremented at method entry. A preserved `RowType`/`ArrayType` can contain descendants that would normally consume additional units, so later Variant columns can now exceed `variant.shredding.maxSchemaWidth`.
   
   I reproduced this with `maxSchemaWidth = 8` and two Variant columns: the first evolves from `p: scalar` to `p: {x: scalar}` and then omits `p`, while the second adds a new `r` field in the third round. This branch carries `ROW<x BIGINT>` forward without accounting for `x`, then also selects `q` and `r` in the second column, using 9 budget units. A focused assertion that `r` remains untyped fails because it becomes `BIGINT`.
   
   Please preserve the selected schema while also walking/debiting its nested width (and handling insufficient remaining budget consistently) before returning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org