[I] [Feature] Build and drop full-text global indexes natively [paimon-rust]

["zhuxiangyi (via GitHub)" <gi...@apache.org>]

zhuxiangyi opened a new issue, #962:
URL: https://github.com/apache/paimon-rust/issues/962

   ### Search before asking
   
   - [x] I searched in the issues and found nothing similar.
   
   ### Motivation
   
   paimon-rust can already search `full-text` global indexes (`FullTextSearchBuilder`,
   `full_text_search`, hybrid search), but it cannot build or drop them:
   
   - `CALL sys.create_global_index(..., index_type => 'full-text')` is rejected. The procedure supports
     only `btree`, `bitmap`, `multivalue`, `fm`, and the vindex types.
   - `CALL sys.drop_global_index(..., index_type => 'full-text')` is rejected as an unsupported type.
   
   So a table written by Rust, pypaimon native, or C FFI needs a separate Java (Flink/Spark) job before
   full-text search can use an index. Until then, every search either returns nothing (`fast` mode)
   or builds a temporary in-memory index over the raw rows (`full`/`detail` mode).
   
   ### Solution
   
   Add a native full-text index build that mirrors Java's generic global-index build
   (`GenericIndexTopoBuilder` driving `NativeFullTextGlobalIndexWriter`):
   
   - Split the latest snapshot's row IDs into `global-index.row-count-per-shard` shards. Skip ranges a
     `full-text` index on the column already covers, so repeated builds are incremental.
   - Feed each shard's text column to `paimon-ftindex-core` with shard-relative row IDs. This is the
     native core the Rust reader already uses. Write one `full-text` index file per non-empty shard
     and commit all shards in one snapshot.
   - Follow Java's semantics:
     - only `full-text.*` options reach the native writer, with the prefix removed;
     - `index_meta` is the flat JSON of those options;
     - NULL rows count toward the row count but are not indexed.
   - Expose it as `Table::new_full_text_index_build_builder()` (behind the `fulltext` feature). Route
     `create_global_index(index_type => 'full-text')` to it in DataFusion, and accept `full-text` in
     `drop_global_index`.
   
   Table requirements would match the existing global-index builders: row tracking, data evolution,
   global index enabled, no primary keys, no deletion vectors. The column must be CHAR/VARCHAR.
   
   ### Anything else?
   
   Out of scope for the first PR, as possible follow-ups:
   
   - allow tables with deletion vectors (Java's generic build does not reject them);
   - refresh indexes on column updates (`global-index.column-update-action`).
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Build and drop full-text global indexes natively [paimon-rust]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #962: [Feature] Build and drop full-text global indexes natively
URL: https://github.com/apache/paimon-rust/issues/962


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org