You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liujiwen-up (via GitHub)" <gi...@apache.org> on 2026/06/04 03:28:41 UTC
[PR] test: add mixed-format drop column schema evolution coverage [paimon-rust]
liujiwen-up opened a new pull request, #363:
URL: https://github.com/apache/paimon-rust/pull/363
### Purpose
Linked issue: #258
This PR adds focused coverage for mixed-format schema evolution after `DROP COLUMN`.
The fixture writes old-schema data files in Parquet and ORC with a dropped column, then drops that column and writes new-schema data files in Avro. The read test verifies that Rust can scan all three formats together while exposing only the remaining table columns.
### Brief change log
* Add `mixed_format_schema_evolution_drop_column` to the Spark provisioning script.
* Write old-schema Paimon data files in Parquet and ORC formats with `(id, name, score)`.
* Drop `score`, switch to Avro, and write new-schema files with `(id, name)`.
* Add a Rust integration test that:
* verifies the scan plan includes `parquet`, `orc`, and `avro` data files;
* reads the table through `FileSystemCatalog`;
* checks that the dropped `score` column is not present in output batches;
* validates decoded `id` and `name` rows across all file formats.
### Tests
* `cargo fmt --all`
* `cargo test -p paimon-integration-tests test_read_mixed_format_schema_evolution_drop_column --no-run`
* `cargo check -p paimon-integration-tests`
* `python3 -m py_compile dev/spark/provision.py`
* `git diff --check -- dev/spark/provision.py crates/integration_tests/tests/read_tables.rs`
Full runtime execution of the new integration test requires regenerating the Spark-provisioned warehouse with `make docker-up`.
### API and Format
No API or persistent format changes.
### Documentation
No documentation changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test: add mixed-format drop column schema evolution coverage [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #363:
URL: https://github.com/apache/paimon-rust/pull/363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test: add mixed-format drop column schema evolution coverage [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #363:
URL: https://github.com/apache/paimon-rust/pull/363#discussion_r3379807135
##########
crates/integration_tests/tests/read_tables.rs:
##########
@@ -1583,6 +1583,109 @@ async fn test_read_schema_evolution_rename_column() {
);
}
+/// Test reading a mixed-format table after ALTER TABLE DROP COLUMN.
+/// Old Parquet/ORC data files have the dropped column; new Avro files do not.
+#[tokio::test]
+async fn test_read_mixed_format_schema_evolution_drop_column() {
+ let (plan, batches) =
+ scan_and_read_with_fs_catalog("mixed_format_schema_evolution_drop_column", None).await;
+
+ let formats: HashSet<&str> = plan
Review Comment:
nit: This manually repeats the same file-format assertion already covered by the `assert_plan_file_formats` helper. Consider using `assert_plan_file_formats(&plan, &["avro", "orc", "parquet"], "mixed_format_schema_evolution_drop_column");` to stay consistent with the existing mixed-format tests and reduce duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org