You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "vamsikarnika (via GitHub)" <gi...@apache.org> on 2026/07/21 06:33:34 UTC

[PR] fix: relax existing column to nullable in reconcileSchema when source made it nullable [hudi]

vamsikarnika opened a new pull request, #19337:
URL: https://github.com/apache/hudi/pull/19337

   ### Change Logs
   
   `AvroSchemaEvolutionUtils.reconcileSchema` — invoked from `HoodieSchemaUtils.deduceWriterSchema` when `hoodie.write.set.null.for.missing.columns=true` — rebuilds the writer schema **from the existing table schema** and only altered a shared column on a *data-type* change (`findType` ignores nullability). As a result, an existing column that the incoming/source schema relaxed from **required → nullable** stayed `REQUIRED` in the table indefinitely, so records arriving with `null` in that column failed record creation (and, where an error table is configured, were routed to the quarantine table).
   
   Note the `set.null.for.missing.columns=false` path already preserved the relaxation (it uses the canonicalized source schema directly); only the `=true` path silently dropped it.
   
   **Fix:** in `reconcileSchema`, detect existing columns whose incoming schema is nullable but the table is required (`nullabilityRelaxColumns`), exclude them from the early-return short-circuit, and relax them to nullable in the result via `updateColumnNullability(col, true)`. This **only ever widens, never tightens** (a nullable→required incoming leaves the table nullable), consistent with `reconcileSchemaRequirements`, and still null-fills genuinely missing columns.
   
   Tests:
   - `TestAvroSchemaEvolutionUtils` — direct `reconcileSchema` relax assertion + a no-tighten guard.
   - `TestHoodieSchemaUtils` — end-to-end `deduceWriterSchema`, parameterized over the flag.
   - Verified both new relax-cases **fail without the change** (`expected optional boolean but was required boolean`); full `TestAvroSchemaEvolutionUtils` (15) + `TestHoodieSchemaUtils` (11) pass with it.
   
   ### Impact
   
   Tables ingesting via the streamer/datasource with `set.null.for.missing.columns=true` will now correctly evolve an existing column to nullable when the source relaxes it, instead of leaving it required and failing/quarantining null-bearing records. Widening-only; no existing column is ever tightened, and missing-column null-filling is unchanged.
   
   ### Risk level: low
   
   Change is confined to `reconcileSchema` and only relaxes nullability of a shared column when the incoming schema is already nullable — a backwards-compatible schema widening. Covered by new unit tests and the existing `AvroSchemaEvolutionUtils` / `HoodieSchemaUtils` suites.
   
   ### Documentation Update
   
   None required — behavior now matches the documented intent of schema evolution (a valid required→nullable relaxation should be applied).
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [x] Adequate tests were added (unit tests, incl. a failing-without-fix check)
   - [x] CI passes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org