You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liwuhen (via GitHub)" <gi...@apache.org> on 2026/07/23 12:13:50 UTC
[PR] fix(table): validate write batch schema at the core entry [paimon-rust]
liwuhen opened a new pull request, #602:
URL: https://github.com/apache/paimon-rust/pull/602
### Purpose
`TableWrite::write_arrow_batch` currently accepts an Arrow `RecordBatch`
without validating that its schema matches the table's write contract.
Malformed batches can therefore reach downstream writers. In particular, the
dedicated BLOB and VECTOR writer assumes that input columns match the table
schema and indexes them directly. A batch with missing fields can panic instead
of returning a normal Paimon error.
The empty-batch fast path also returns successfully before any downstream
component can notice an invalid schema.
This change validates the batch schema at the public table-write entry before
the empty-batch fast path, row-kind enrichment, partition routing, or writer
dispatch.
Validation is performed in the following order:
- Field count.
- Field names and order.
- Arrow data types.
The accepted schemas are:
- Normal writes accept exactly the table fields.
- `changelog-producer=input` accepts either the table fields or the table
fields followed by `_VALUE_KIND: Int8`.
- Cross-partition and `rowkind.field` writes continue to accept only the table
fields. Their internal `_VALUE_KIND` generation remains unchanged.
Every schema mismatch returns `Error::DataInvalid` with expected and actual
context for the first mismatch. No casting, field reordering, column insertion,
or schema coercion is performed.
### Brief change log
- Cache the table's expected Arrow write schema in `TableWrite` using the
existing `build_target_arrow_schema` conversion.
- Add a private schema validator to `TableWrite::write_arrow_batch`.
- Run validation before the empty-batch fast path and writer dispatch.
- Validate field count before names/order, and names/order before Arrow data
types.
- Allow an optional trailing `_VALUE_KIND: Int8` only for
`changelog-producer=input`.
- Reuse the cached Arrow schema when creating the dedicated-format writer.
- Add regression coverage for:
- Missing BLOB fields.
- Missing VECTOR fields.
- Reordered fields.
- Incorrect field names.
- Incorrect Arrow data types.
- Unexpected extra columns.
- `_VALUE_KIND` on normal writes.
- Misplaced or incorrectly typed `_VALUE_KIND` on input changelog writes.
- Invalid schemas on empty batches.
- Caller-provided `_VALUE_KIND` in cross-partition and `rowkind.field`
modes.
- Valid normal, input changelog, cross-partition, and `rowkind.field`
writes.
### Tests
- Unit test `test_write_arrow_batch_rejects_missing_blob_field`
(`table_write.rs`): verifies that a BLOB batch with a missing field returns
`DataInvalid` instead of reaching the dedicated writer and panicking.
**Fails on the pre-fix code.**
- Unit test `test_write_arrow_batch_rejects_missing_vector_field`
(`table_write.rs`): verifies the same behavior for a dedicated VECTOR
writer. **Fails on the pre-fix code.**
- Unit test `test_write_arrow_batch_rejects_mismatched_table_fields`
(`table_write.rs`): covers field reordering, incorrect names, incorrect
Arrow types, and unexpected columns.
- Unit test `test_write_arrow_batch_rejects_invalid_empty_batch_schema`
(`table_write.rs`): verifies that schema validation occurs before the empty
batch fast path. **Fails on the pre-fix code.**
- Unit test `test_normal_write_rejects_appended_value_kind`
(`table_write.rs`): verifies that normal writes reject a caller-provided
`_VALUE_KIND`.
- Unit test
`test_input_changelog_rejects_misplaced_or_wrong_type_value_kind`
(`table_write.rs`): verifies that input changelog writes accept
`_VALUE_KIND` only as the final `Int8` field.
- Unit test
`test_rowkind_field_accepts_table_schema_and_rejects_caller_value_kind`
(`table_write.rs`): verifies that `rowkind.field` retains its existing
caller schema contract and internal value-kind generation.
- Unit test `test_cross_partition_write_rejects_caller_value_kind`
(`table_write.rs`): verifies that cross-partition writes continue to accept
only table fields.
- Existing table-write tests verify valid normal, input changelog, and
cross-partition writes.
- Commands run locally:
- `cargo test -p paimon --lib table_write`
- `cargo test -p paimon --lib`
- `cargo fmt --all --check`
- `cargo check -p paimon --all-targets`
- `cargo clippy -p paimon --all-targets -- -D warnings`
- `git diff --check`
### API and Format
No public API or persisted data format changes.
The change only rejects Arrow batch schemas that do not satisfy the existing
table-write contract. It does not cast, reorder, pad, or coerce input columns.
Top-level Arrow field nullability and field metadata are not added to the
validation contract. Existing schema evolution behavior is unchanged.
Changelog, cross-partition, and `rowkind.field` semantics remain unchanged.
The dedicated BLOB and VECTOR formats and their downstream writing logic are
not modified.
### Documentation
No documentation changes required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #602:
URL: https://github.com/apache/paimon-rust/pull/602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #602:
URL: https://github.com/apache/paimon-rust/pull/602#discussion_r3646781219
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -451,6 +456,107 @@ impl TableWrite {
Ok(())
}
+ fn validate_write_batch_schema(&self, batch: &RecordBatch) -> Result<()> {
+ let expected_schema = &self.write_schema;
+ let actual_schema = batch.schema();
+ let table_field_count = expected_schema.fields().len();
+ let actual_field_count = actual_schema.fields().len();
+ // Cross-partition routing generates its own `_VALUE_KIND` column so that
+ // migrated rows can be written as deletes in their previous partition.
+ // Accepting a caller-provided column here would make the generated delete
+ // batch contain `_VALUE_KIND` twice.
+ let allows_value_kind = self.changelog_producer == ChangelogProducer::Input
Review Comment:
This condition conflates input row kinds with changelog-file production. `_VALUE_KIND` is consumed by the data writer even when `changelog-producer=none`, while `ChangelogProducer::Input` only controls writing separate changelog files. As a result, primary-key tables using the default producer now reject valid DELETE/UPDATE batches before they reach the writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #602:
URL: https://github.com/apache/paimon-rust/pull/602#discussion_r3642472534
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -451,6 +456,102 @@ impl TableWrite {
Ok(())
}
+ fn validate_write_batch_schema(&self, batch: &RecordBatch) -> Result<()> {
+ let expected_schema = &self.write_schema;
+ let actual_schema = batch.schema();
+ let table_field_count = expected_schema.fields().len();
+ let actual_field_count = actual_schema.fields().len();
+ let allows_value_kind = self.changelog_producer == ChangelogProducer::Input;
Review Comment:
`changelog-producer=input` is also permitted on cross-partition tables, so this condition accepts a caller-provided `_VALUE_KIND` in that mode. When a row moves partitions, the routing path appends a second `_VALUE_KIND` for the generated delete, causing a schema/column-count error in the KV writer. The new entry validation therefore still accepts a batch that fails downstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org