You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2026/09/30 04:39:46 UTC

[I] [Feature] Support table-level full-text search for data-evolution tables [paimon-cpp]

SteNicholas opened a new issue, #404:
URL: https://github.com/apache/paimon-cpp/issues/404

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar.
   
   ### Motivation
   
   Sub-issue of #399 (step 4: table-level full-text search).
   
   Paimon C++ only exposes the reader primitive. To run a full-text search, a caller has to:
   
   1. call `GlobalIndexScan::CreateReaders`;
   2. call `VisitFullTextSearch` on the readers;
   3. pass the result to `ScanContextBuilder::SetGlobalIndexResult`.
   
   Along the way the caller also has to choose index types and shard ranges, and there is no serializable plan for running scan and read in different processes. Java offers a table-level API instead:
   
   ```java
   GlobalIndexResult result =
           table.newFullTextSearchBuilder()
                   .withQuery("content", "{\"match\":{\"query\":\"paimon lake\"}}")
                   .withLimit(10)
                   .executeLocal();
   TableScan.Plan plan = table.newReadBuilder().newScan().withGlobalIndexResult(result).plan();
   ```
   
   The Java components live in paimon-core `org.apache.paimon.table.source`, plus one method in paimon-common:
   
   | Java class / method | What it provides |
   |---|---|
   | `FullTextSearchBuilder` / `FullTextSearchBuilderImpl` | `withPartitionFilter`, `withFilter` (see {{S6}}), `withLimit`, `withQuery(fieldName, query)`, `newFullTextScan()`, `newFullTextRead()`, `executeLocal()`. A package-private `withSnapshot` is used by hybrid search. The query column must exist, and `newFullTextRead` requires `limit > 0`. |
   | `FullTextScan#scan()` | Returns `Plan { splits(), snapshot() }`; implemented by `DataEvolutionFullTextScan`. |
   | `FullTextRead#read(plan \| splits)` | Returns a `GlobalIndexResult`; implemented by `DataEvolutionFullTextRead`. |
   | `FullTextSearchSplit`, `IndexFullTextSearchSplit` | The split carries the column, the physical `rowRangeStart`/`rowRangeEnd`, the assigned `searchRowRanges`, the full-text index files and the scalar index files, with versioned serialization. |
   | `GlobalIndexerFactory#supportsFullTextSearch()` (paimon-common) | Defaults to `false` and is `true` for `full-text` (apache/paimon#8308). The scan uses it to decide which index files can serve full-text search. |
   
   ### Solution
   
   1. **API.** Add C++ equivalents of the builder, scan, read and splits, following the existing builder conventions (`ScanContextBuilder` / `ReadContextBuilder`). Splits must be serializable so scan and read can run in different processes. The read result plugs into `ScanContextBuilder::SetGlobalIndexResult`, and scores stay readable through `_INDEX_SCORE`.
   2. **Capability.** Add `SupportsFullTextSearch()` to `GlobalIndexerFactory` or `GlobalIndexer`, returning `true` for `full-text`. Decide whether `lucene-fts` returns `true` as well.
   3. **Scan**, following `DataEvolutionFullTextScan`:
      - Pin the snapshot: the latest one, or the time-travel target.
      - Keep index manifest entries that have `GlobalIndexMeta` and pass the partition filter.
      - Treat a file as a full-text input when the column is its `index_field_id` or one of its `extra_field_ids`, and its factory supports full-text search.
      - Select ranges across overlapping and multi-field indexes (apache/paimon#8547):
        1. Group candidates by column → physical range → index identity, where the identity is `type|index_field_id|extra_field_ids`.
        2. Sweep the elementary intervals and give each one to the best active candidate. A dedicated index comes first, then order by identity, `from` and `to`.
        3. Merge adjacent intervals.
        4. Emit one `IndexFullTextSearchSplit` per selection. It keeps the physical range, which is needed for the row-id offset, and the assigned `searchRowRanges`.
   4. **Read**, following `DataEvolutionFullTextRead`:
      - Run on an executor sized by `global-index.thread-num`.
      - For each split:
        - Create the indexer from the split's own file meta, including any extra fields, and wrap its reader in `OffsetGlobalIndexReader(start, end)`.
        - Search with `FullTextSearch(column, query, limit)`, restricted to `searchRowRanges ∧ liveRows`. Pass no restriction when this covers the whole physical range, and skip the split when it is empty.
        - Close the reader afterwards.
      - Merge the per-split results, keeping the first score for a duplicate row id. Then apply the final `topK(limit)`, ordered by score descending and then row id ascending. Each split only needs `limit` candidates (apache/paimon#9955).
      - Pin the live-row computation ({{S4}}) and all reads to the plan snapshot (apache/paimon#9031).
   5. **Tests.** Cover:
      - multiple shards
      - overlapping index ranges
      - a text column that is an extra field of another index
      - partition filters
      - snapshot pinning
      - deletion-vector tables
      - a split serialization round trip
   
   ### Anything else?
   
   - Depends on {{S1}}, {{S2}}, {{S3}} and {{S4}}.
   - Java's `IndexFullTextSearchSplit` uses Java serialization, so there is no cross-language wire format to match. C++ should define its own versioned serialization.
   - Java rejects search on tables under query authorization (apache/paimon#8570). C++ should do the same if it adds query auth.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Support table-level full-text search for data-evolution tables [paimon-cpp]

Posted by "SteNicholas (via GitHub) via issues" <is...@paimon.apache.org>.
SteNicholas commented on issue #404:
URL: https://github.com/apache/paimon-cpp/issues/404#issuecomment-6569752900

   Follow-up after #461: since it was merged, the `tsan-x86_64` job of `Build and Test` runs past its 120-minute timeout ([run 41000500442](https://github.com/apache/paimon-cpp/actions/runs/41000500442/job/123883528750)). `paimon-global-index-test` grew from about 1400s to about 3500s on `tsan-x86_64`, and from about 1000s to about 3000s on `tsan-aarch64`.
   
   **Root cause:** every `LuceneGlobalIndexReader` and `LuceneGlobalIndexWriter` constructed its own `cppjieba::Jieba`, reloading about 11.6MB of Jieba dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard, and the table-level full-text search tests added in #461 run many such queries. The cost is not limited to tests: every table-level search on a `lucene-fts` index pays it in production too.
   
   **Fix:** #486 loads each dictionary directory once per process and shares a const segmenter with all Lucene index readers and writers. With it, `paimon-global-index-test` takes about 156s on `tsan-x86_64` and about 103s on `tsan-aarch64`, and the `tsan-x86_64` job passes in about an hour.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Support table-level full-text search for data-evolution tables [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 closed issue #404: [Feature] Support table-level full-text search for data-evolution tables
URL: https://github.com/apache/paimon-cpp/issues/404


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org