You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "QuakeWang (via GitHub)" <gi...@apache.org> on 2026/07/06 13:28:05 UTC
[PR] fix(core): validate writer batch schema [paimon-mosaic]
QuakeWang opened a new pull request, #70:
URL: https://github.com/apache/paimon-mosaic/pull/70
MosaicWriter builds `batch_col_map` from the initial Arrow schema, but `write_batch` previously only checked column count and non-nullable null counts. A later batch with the same number of same-typed columns but reordered field names could be accepted and written through the stale column map, silently storing values under the wrong column names.
This patch validates each incoming batch against the writer schema before using `batch_col_map`: field name, logical type, and nullability must match. It also adds a regression test for the same-typed reordered-column case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): validate writer batch schema [paimon-mosaic]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on PR #70:
URL: https://github.com/apache/paimon-mosaic/pull/70#issuecomment-5750102753
For the 0.3.0 release, I propose treating this as a correctness blocker: a reordered batch with compatible types can otherwise be written under the wrong logical columns without an error. The current head is mergeable and all five CI jobs are green.
@JingsongLi could you please review this for inclusion in 0.3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): validate writer batch schema [paimon-mosaic]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on PR #70:
URL: https://github.com/apache/paimon-mosaic/pull/70#issuecomment-5354050237
The follow-up changes from QuakeWang/paimon-mosaic#1 have been merged into `75eeeffe`.
The final scope is limited to validating mapped top-level field names before writing, which prevents same-typed reordered batches from being routed to the wrong logical columns. The existing writer validation continues to handle Arrow array types and actual null values.
All five CI jobs are green: https://github.com/apache/paimon-mosaic/actions/runs/32353562910.
The PR is ready for another review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): validate writer batch schema [paimon-mosaic]
Posted by "jianguotian (via GitHub)" <gi...@apache.org>.
jianguotian commented on PR #70:
URL: https://github.com/apache/paimon-mosaic/pull/70#issuecomment-5336933071
I found and fixed two regressions in the current head:
- keep field-name/order validation, but leave type/nullability enforcement on the existing writer validation path so current error/test contracts remain intact;
- report the actual mapped batch column index in the name-mismatch error.
Because I cannot push to `QuakeWang:mosaic-schema-guard`, the fix is available as `fbaf0a0b5df19f5d7141e4b78ccfbc17601c1215` and helper PR https://github.com/QuakeWang/paimon-mosaic/pull/1.
Validation:
- local full Rust/Python/C++/Java suites, fmt and Clippy passed;
- exact-head GitHub CI against current Apache main `b613dd25eba45afbc860171e109df19495ba0e3c`: all 5 jobs passed: https://github.com/jianguotian/paimon-mosaic/actions/runs/32209856324.
Please merge the helper PR (or cherry-pick the commit) to update this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(core): validate writer batch schema [paimon-mosaic]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #70:
URL: https://github.com/apache/paimon-mosaic/pull/70
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org