You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "liujiwen-up (via GitHub)" <gi...@apache.org> on 2026/06/16 15:54:06 UTC
[PR] feat(datafusion): add Lumina index build procedure [paimon-rust]
liujiwen-up opened a new pull request, #392:
URL: https://github.com/apache/paimon-rust/pull/392
### Purpose
Linked issue: none.
Expose the core Lumina index build capability through DataFusion SQL so users can build a Lumina global vector index with a `CALL sys.create_lumina_index(...)` statement, then query it through the existing `vector_search` table function.
### Brief change log
- Add `CALL sys.create_lumina_index(table => ..., index_column => ...)` to DataFusion procedures.
- Wire optional Lumina builder settings through `index_type` and comma-separated `options => 'key=value,...'` arguments.
- Add procedure error coverage for missing `index_column`, invalid index type, and malformed options.
- Add an ignored native Lumina E2E test that writes vector data, builds the Lumina index via SQL, verifies `$table_indexes`, and runs `vector_search`.
- Add a CI step to run the native Lumina DataFusion E2E test with the `vortex` feature after installing the Lumina native library.
- Document the new SQL procedure and options syntax.
### Tests
- `cargo fmt`: passed
- `cargo check -p paimon-datafusion`: passed
- `cargo test -p paimon-datafusion --test procedures create_lumina_index`: passed, 3 tests
- `cargo test -p paimon-datafusion --features vortex vector_search_tests::test_lumina_build_then_vector_search_query -- --ignored --exact --list`: passed, verifies the ignored native E2E test compiles and is discoverable
### API and Format
Adds a DataFusion SQL procedure surface for an existing core Lumina index build API. No storage format, manifest format, or public Rust API changes are introduced by this PR.
### Documentation
Updated `docs/src/sql.md` with the `create_lumina_index` procedure and optional options example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Lumina index build procedure [paimon-rust]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #392:
URL: https://github.com/apache/paimon-rust/pull/392#discussion_r3443273694
##########
docs/src/sql.md:
##########
@@ -419,6 +419,24 @@ Rollback a table to a specific timestamp:
CALL sys.rollback_to_timestamp(table => 'paimon.my_db.my_table', timestamp => 1234567890000);
```
+### create_lumina_index
Review Comment:
thanks, i done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Lumina index build procedure [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on PR #392:
URL: https://github.com/apache/paimon-rust/pull/392#issuecomment-4755767433
Please rebase main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Lumina index build procedure [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #392:
URL: https://github.com/apache/paimon-rust/pull/392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Lumina index build procedure [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #392:
URL: https://github.com/apache/paimon-rust/pull/392#discussion_r3425681062
##########
docs/src/sql.md:
##########
@@ -419,6 +419,24 @@ Rollback a table to a specific timestamp:
CALL sys.rollback_to_timestamp(table => 'paimon.my_db.my_table', timestamp => 1234567890000);
```
+### create_lumina_index
Review Comment:
Nit: the procedure also accepts the optional `index_type` argument, while the docs only mention `options`. It may be worth documenting the default and valid values here so the SQL surface matches the implemented arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(datafusion): add Lumina index build procedure [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #392:
URL: https://github.com/apache/paimon-rust/pull/392#discussion_r3451228432
##########
docs/src/sql.md:
##########
@@ -419,6 +419,35 @@ Rollback a table to a specific timestamp:
CALL sys.rollback_to_timestamp(table => 'paimon.my_db.my_table', timestamp => 1234567890000);
```
+### create_lumina_index
+
+Build and commit a Lumina global vector index for a table column:
+
+```sql
+CALL sys.create_lumina_index(table => 'paimon.my_db.my_table', index_column => 'embedding');
+```
+
+The optional `index_type` argument selects the Lumina index identifier. It defaults to
Review Comment:
Non-blocking: this wording suggests `index_type` controls the index identifier written to the manifest. The builder accepts `lumina-vector-ann` for compatibility, but newly built indexes are canonicalized and stored as `lumina`. Could we clarify that here to avoid exposing a misleading contract?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org