You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 09:09:31 UTC
[I] Fail the delta streamer pipeline on re-ordering of columns [hudi]
hudi-bot opened a new issue, #16266:
URL: https://github.com/apache/hudi/issues/16266
Re-ordering of columns are not supported in delta streamer code path. We need to detect and fail the pipeline processing. Currently it will continue ingestion and fail during read side. We should be proactive and fail during ingestion.
Parent PR related to this issue - https://github.com/apache/hudi/pull/9743
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6955
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fail the delta streamer pipeline on re-ordering of columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16266: Fail the delta streamer pipeline on re-ordering of columns
URL: https://github.com/apache/hudi/issues/16266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fail the delta streamer pipeline on re-ordering of columns [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16266:
URL: https://github.com/apache/hudi/issues/16266#issuecomment-4795566678
Triage note: this appears to have been addressed by [HUDI-7713 / PR #11154](https://github.com/apache/hudi/pull/11154) (commit `d964895`), "Enforce ordering of fields during schema reconciliation."
In current `master`, `HoodieSchemaUtils.canonicalizeSchema` (used by the DeltaStreamer/Spark write path) calls `AvroSchemaEvolutionUtils.reconcileSchemaRequirements(source, target, shouldReorderColumns = !RECONCILE_SCHEMA)`. With the default `RECONCILE_SCHEMA=false`, the source schema's field positions are realigned to the table schema's positions via `NameToPositionVisitor` before writing, so a reordered incoming batch no longer produces files that fail at read time.
The resolution differs from this issue's original proposal (silent reorder rather than proactive failure), but the underlying read-side failure mode should no longer be reachable. If you have a current repro that still hits a read-time failure on `master`, please share it and we can reopen — otherwise this can likely be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org