You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liwuhen (via GitHub)" <gi...@apache.org> on 2026/07/23 12:13:50 UTC

[PR] fix(table): validate write batch schema at the core entry [paimon-rust]

liwuhen opened a new pull request, #602:
URL: https://github.com/apache/paimon-rust/pull/602

   ### Purpose
   
   `TableWrite::write_arrow_batch` currently accepts an Arrow `RecordBatch`
   without validating that its schema matches the table's write contract.
   
   Malformed batches can therefore reach downstream writers. In particular, the
   dedicated BLOB and VECTOR writer assumes that input columns match the table
   schema and indexes them directly. A batch with missing fields can panic instead
   of returning a normal Paimon error.
   
   The empty-batch fast path also returns successfully before any downstream
   component can notice an invalid schema.
   
   This change validates the batch schema at the public table-write entry before
   the empty-batch fast path, row-kind enrichment, partition routing, or writer
   dispatch.
   
   Validation is performed in the following order:
   
   - Field count.
   - Field names and order.
   - Arrow data types.
   
   The accepted schemas are:
   
   - Normal writes accept exactly the table fields.
   - `changelog-producer=input` accepts either the table fields or the table
     fields followed by `_VALUE_KIND: Int8`.
   - Cross-partition and `rowkind.field` writes continue to accept only the table
     fields. Their internal `_VALUE_KIND` generation remains unchanged.
   
   Every schema mismatch returns `Error::DataInvalid` with expected and actual
   context for the first mismatch. No casting, field reordering, column insertion,
   or schema coercion is performed.
   
   ### Brief change log
   
   - Cache the table's expected Arrow write schema in `TableWrite` using the
     existing `build_target_arrow_schema` conversion.
   - Add a private schema validator to `TableWrite::write_arrow_batch`.
   - Run validation before the empty-batch fast path and writer dispatch.
   - Validate field count before names/order, and names/order before Arrow data
     types.
   - Allow an optional trailing `_VALUE_KIND: Int8` only for
     `changelog-producer=input`.
   - Reuse the cached Arrow schema when creating the dedicated-format writer.
   - Add regression coverage for:
     - Missing BLOB fields.
     - Missing VECTOR fields.
     - Reordered fields.
     - Incorrect field names.
     - Incorrect Arrow data types.
     - Unexpected extra columns.
     - `_VALUE_KIND` on normal writes.
     - Misplaced or incorrectly typed `_VALUE_KIND` on input changelog writes.
     - Invalid schemas on empty batches.
     - Caller-provided `_VALUE_KIND` in cross-partition and `rowkind.field`
       modes.
     - Valid normal, input changelog, cross-partition, and `rowkind.field`
       writes.
   
   ### Tests
   
   - Unit test `test_write_arrow_batch_rejects_missing_blob_field`
     (`table_write.rs`): verifies that a BLOB batch with a missing field returns
     `DataInvalid` instead of reaching the dedicated writer and panicking.
     **Fails on the pre-fix code.**
   - Unit test `test_write_arrow_batch_rejects_missing_vector_field`
     (`table_write.rs`): verifies the same behavior for a dedicated VECTOR
     writer. **Fails on the pre-fix code.**
   - Unit test `test_write_arrow_batch_rejects_mismatched_table_fields`
     (`table_write.rs`): covers field reordering, incorrect names, incorrect
     Arrow types, and unexpected columns.
   - Unit test `test_write_arrow_batch_rejects_invalid_empty_batch_schema`
     (`table_write.rs`): verifies that schema validation occurs before the empty
     batch fast path. **Fails on the pre-fix code.**
   - Unit test `test_normal_write_rejects_appended_value_kind`
     (`table_write.rs`): verifies that normal writes reject a caller-provided
     `_VALUE_KIND`.
   - Unit test
     `test_input_changelog_rejects_misplaced_or_wrong_type_value_kind`
     (`table_write.rs`): verifies that input changelog writes accept
     `_VALUE_KIND` only as the final `Int8` field.
   - Unit test
     `test_rowkind_field_accepts_table_schema_and_rejects_caller_value_kind`
     (`table_write.rs`): verifies that `rowkind.field` retains its existing
     caller schema contract and internal value-kind generation.
   - Unit test `test_cross_partition_write_rejects_caller_value_kind`
     (`table_write.rs`): verifies that cross-partition writes continue to accept
     only table fields.
   - Existing table-write tests verify valid normal, input changelog, and
     cross-partition writes.
   - Commands run locally:
     - `cargo test -p paimon --lib table_write`
     - `cargo test -p paimon --lib`
     - `cargo fmt --all --check`
     - `cargo check -p paimon --all-targets`
     - `cargo clippy -p paimon --all-targets -- -D warnings`
     - `git diff --check`
   
   ### API and Format
   
   No public API or persisted data format changes.
   
   The change only rejects Arrow batch schemas that do not satisfy the existing
   table-write contract. It does not cast, reorder, pad, or coerce input columns.
   
   Top-level Arrow field nullability and field metadata are not added to the
   validation contract. Existing schema evolution behavior is unchanged.
   
   Changelog, cross-partition, and `rowkind.field` semantics remain unchanged.
   The dedicated BLOB and VECTOR formats and their downstream writing logic are
   not modified.
   
   ### Documentation
   
   No documentation changes required.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #602:
URL: https://github.com/apache/paimon-rust/pull/602


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]

Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #602:
URL: https://github.com/apache/paimon-rust/pull/602#discussion_r3646781219


##########
crates/paimon/src/table/table_write.rs:
##########
@@ -451,6 +456,107 @@ impl TableWrite {
         Ok(())
     }
 
+    fn validate_write_batch_schema(&self, batch: &RecordBatch) -> Result<()> {
+        let expected_schema = &self.write_schema;
+        let actual_schema = batch.schema();
+        let table_field_count = expected_schema.fields().len();
+        let actual_field_count = actual_schema.fields().len();
+        // Cross-partition routing generates its own `_VALUE_KIND` column so that
+        // migrated rows can be written as deletes in their previous partition.
+        // Accepting a caller-provided column here would make the generated delete
+        // batch contain `_VALUE_KIND` twice.
+        let allows_value_kind = self.changelog_producer == ChangelogProducer::Input

Review Comment:
   This condition conflates input row kinds with changelog-file production. `_VALUE_KIND` is consumed by the data writer even when `changelog-producer=none`, while `ChangelogProducer::Input` only controls writing separate changelog files. As a result, primary-key tables using the default producer now reject valid DELETE/UPDATE batches before they reach the writer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(table): validate write batch schema at the core entry [paimon-rust]

Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #602:
URL: https://github.com/apache/paimon-rust/pull/602#discussion_r3642472534


##########
crates/paimon/src/table/table_write.rs:
##########
@@ -451,6 +456,102 @@ impl TableWrite {
         Ok(())
     }
 
+    fn validate_write_batch_schema(&self, batch: &RecordBatch) -> Result<()> {
+        let expected_schema = &self.write_schema;
+        let actual_schema = batch.schema();
+        let table_field_count = expected_schema.fields().len();
+        let actual_field_count = actual_schema.fields().len();
+        let allows_value_kind = self.changelog_producer == ChangelogProducer::Input;

Review Comment:
   `changelog-producer=input` is also permitted on cross-partition tables, so this condition accepts a caller-provided `_VALUE_KIND` in that mode. When a row moves partitions, the routing path appends a second `_VALUE_KIND` for the generated delete, causing a schema/column-count error in the KV writer. The new entry validation therefore still accepts a batch that fails downstream.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org