You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liujiwen-up (via GitHub)" <gi...@apache.org> on 2026/05/14 15:32:57 UTC
[PR] test(datafusion): add Vortex SQL E2E test [paimon-rust]
liujiwen-up opened a new pull request, #321:
URL: https://github.com/apache/paimon-rust/pull/321
### Purpose
Linked issue: close #xxx
Add a minimal DataFusion SQL end-to-end test for Vortex file format tables.
This verifies that `paimon-datafusion` can enable the core `paimon/vortex` feature, create a table with `file.format = 'vortex'`, insert rows through SQL, and read them back correctly.
### Brief change log
- Add `vortex = ["paimon/vortex"]` feature passthrough to `paimon-datafusion`.
- Add a feature-gated DataFusion integration test for:
- `CREATE TABLE ... WITH ('file.format' = 'vortex')`
- `INSERT INTO`
- `SELECT ... ORDER BY`
- simple `WHERE id = 2` query
- Assert that the SQL flow writes an actual `.vortex` data file.
- Make Vortex reader convert arrays using the target Arrow schema instead of the preferred Arrow representation, so string columns are returned as schema-compatible `Utf8`.
- Add a core-level Vortex reader test covering `STRING` / `Utf8` schema compatibility.
### Tests
- `cargo fmt --all -- --check`
- `cargo check -p paimon-datafusion --features vortex`
- `cargo test -p paimon-datafusion --test vortex_tables --features vortex test_vortex_file_format_sql_e2e`
- `cargo test -p paimon --features vortex test_vortex_reader_returns_utf8_for_string_schema`
- `cargo test -p paimon --features vortex arrow::format::vortex`
### API and Format
No public API change.
No storage format change. This only exposes the existing optional Vortex support through the DataFusion integration feature and fixes Arrow schema compatibility when reading Vortex data.
### Documentation
No documentation update required. The change adds test coverage for existing Vortex support rather than introducing a new user-facing feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Vortex SQL E2E test [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #321:
URL: https://github.com/apache/paimon-rust/pull/321#discussion_r3242869903
##########
crates/integrations/datafusion/tests/vortex_tables.rs:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#![cfg(feature = "vortex")]
Review Comment:
This e2e is currently not exercised by CI because it is behind `#![cfg(feature = "vortex")]`, while the DataFusion CI does not enable the `vortex` feature. Please add a targeted CI step for this test, e.g. `cargo test -p paimon-datafusion --features vortex --test vortex_tables`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Vortex SQL E2E test [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #321:
URL: https://github.com/apache/paimon-rust/pull/321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org