You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/10 11:11:36 UTC
[PR] [docs] Reorganize multimodal table guides and diagrams [paimon]
JingsongLi opened a new pull request, #9721:
URL: https://github.com/apache/paimon/pull/9721
### Purpose
The multimodal table guides mix introductory workflows, storage formats, maintenance, and index reference material in long pages. Reorganize them into a quick start, storage guides, Data Evolution usage/layout/maintenance, and global-index selection/management/reference pages. Preserve existing page URLs and all 101 original section anchors, and add seven SVG diagrams for storage modes, video packs, index coverage, hybrid search, bitmap layout, and Variant shredding.
Make examples consistent with the current implementations: share populated index sample data, separate three-dimensional vector examples from advanced builds, complete BLOB and Variant workflows, and explain scored-result ordering. Clarify engine differences for BLOB updates, video descriptors, index refresh, and visibility callbacks. The quick start now demonstrates why an append needs another index build; coverage modes are explicitly distinguished from freshness after an indexed value changes.
### Tests
- `cd docs && COREPACK_ENABLE_AUTO_PIN=0 yarn build` passed, including REST OpenAPI contract validation.
- Validated 1,043 internal links/fragments in rendered multimodal pages and preservation of all 101 original section anchors.
- Parsed all 49 Python examples and seven SVGs; checked sample Arrow schemas/data, score ordering, and Variant JSON paths. Visually inspected the SVG diagrams in the browser.
- `git diff --check` passed.
- Spark/Flink and native vector/full-text examples were not executed. The targeted Python scalar-index mode test could not start because the local environment lacks `requests`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Reorganize multimodal table guides and diagrams [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9721:
URL: https://github.com/apache/paimon/pull/9721#discussion_r3978755224
##########
docs/docs/multimodal-table/blob.mdx:
##########
@@ -276,72 +162,14 @@ schema = Schema.from_pyarrow_schema(
'data-evolution.enabled': 'true',
}
)
+catalog.create_table('my_db.image_table', schema, False)
Review Comment:
[P2] Keep the add-column examples in the newly created database
This new setup creates `my_db.image_table`, but the Python example under **Adding a Blob Column** still calls `catalog.alter_table('default.image_table', ...)`. Following the walkthrough on a fresh catalog therefore raises `TableNotExistException: Table default.image_table does not exist`. The same mismatch occurs in `vector.mdx`: its setup creates `my_db.vector_table`, while **Adding a Vector Column** alters `default.vector_table`. I executed both create/alter sequences and verified that changing the ALTER targets to their respective `my_db` identifiers succeeds. Please align the identifiers across both walkthroughs.
##########
docs/docs/multimodal-table/global-index.mdx:
##########
@@ -27,425 +26,203 @@ under the License.
# Global Index
-## Overview
+<span id="overview" />
-Global Index is a powerful indexing mechanism for Data Evolution (append) tables. It enables efficient row-level lookups and filtering
-without full-table scans. Paimon supports multiple global index types:
+Global indexes map predicates or search requests to row IDs, which Paimon uses to
+read matching table rows. Choose scalar indexes for filtering, vector indexes for
+similarity search, and full-text indexes for scored text retrieval. Hybrid search
+combines scored routes; it is a query API rather than another stored index type.
-- **[BTree Index](./global-index/btree)**: A B-tree based index for scalar column lookups. Supports equality, IN, range predicates, and can be combined across multiple columns with AND/OR logic.
-- **[Bitmap Index](./global-index/bitmap)**: A bitmap based index for enum-like scalar dimensions and tag columns. Supports equality, IN, prefix match on string columns, complement predicates, and null checks with compressed row-id bitmaps.
-- **[Multivalue Index](./global-index/multivalue)**: A bitmap-backed index for element-membership predicates on `ARRAY` columns.
-- **[FM Index](./global-index/fm)**: An exact partitioned substring index for `CONTAINS` predicates on character columns.
-- **[Vector Index](./global-index/vector)**: An approximate nearest neighbor (ANN) index powered by Paimon's vector index library for vector similarity search.
-- **[Full-Text Index](./global-index/full-text)**: A full-text search index backed by the native full-text engine for text retrieval. Supports term matching and relevance scoring.
-- **[Hybrid Search](./global-index/hybrid-search)**: A multi-route search API that combines results from multiple vector routes, multiple full-text routes, or both before reading table rows.
+This guide covers Data Evolution append tables. Primary-key tables have a separate
+[index configuration and lifecycle](../primary-key-table/global-index).
+
+## Choose an Index
| Index Type | Best For | Notes |
|---|---|---|
-| BTree | Scalar filters on numeric, string, date, and timestamp columns | Best when predicates are selective, such as equality, IN, range, and null checks. |
-| Bitmap | Enum-like dimensions and tag columns | Best for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps. |
-| Multivalue | Membership tests on arrays of supported scalar elements | Best for `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL`. |
-| FM | Exact substring filters on character columns | Supports needles of any byte length and partitions the indexed text for bounded-memory construction and demand-loaded reads. |
-| Vector | Top-K similarity search on embeddings | Uses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size. |
-| Full-Text | Keyword search over text columns | Uses full-text scoring and tokenizer configuration stored with each index file. |
-| Hybrid Search | Combining multiple vector routes, multiple full-text routes, or vector and full-text retrieval together | Runs multiple scored routes and merges them with a ranker before reading rows. |
+| [BTree](./global-index/btree) | Scalar filters on numeric, string, date, and timestamp columns | Best when predicates are selective, such as equality, IN, range, and null checks. |
+| [Bitmap](./global-index/bitmap) | Enum-like dimensions and tag columns | Best for equality, IN, string prefix match, complement predicates, and null checks over compressed row-id bitmaps. |
+| [Multivalue](./global-index/multivalue) | Membership tests on arrays of supported scalar elements | Best for `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL`. |
+| [FM](./global-index/fm) | Exact substring filters on character columns | Supports needles of any byte length and partitions the indexed text for bounded-memory construction and demand-loaded reads. |
+| [Vector](./global-index/vector) | Top-K similarity search on embeddings | Uses ANN algorithms. Tune build-time and search-time options to balance recall, latency, and index size. |
+| [Full-Text](./global-index/full-text) | Keyword search over text columns | Uses full-text scoring and tokenizer configuration stored with each index file. |
+| [Hybrid Search](./global-index/hybrid-search) | Combining multiple vector routes, multiple full-text routes, or vector and full-text retrieval together | Runs multiple scored routes and merges them with a ranker before reading rows. |
-Global indexes work on top of Data Evolution tables. To use global indexes, your table **must** have:
+:::caution Index coverage affects results
-- `'row-tracking.enabled' = 'true'`
-- `'data-evolution.enabled' = 'true'`
+The default search mode is `fast`. If an index covers only part of the table,
+matching rows outside that coverage can be omitted. Creating a table or appending
+data does not build a global index. Read [Coverage and Freshness](./global-index/manage-indexes#coverage-and-freshness)
+before using indexed queries on changing data.
-> Global index queries may not be exact when the index only covers part of the table data. If a query predicate matches the index, Paimon returns only the results from the indexed portion. Matching records in data that has not been indexed yet will not be returned.
+:::
## Prerequisites
-Create a table with the required properties:
+Start with a [configured Paimon catalog](./quick-start). Data Evolution requires
+an append table with row tracking enabled. The following examples assume the `db`
+database exists and that SQL uses that database. Load data before building indexes.
+
+`global-index.enabled` defaults to `true`; setting it does not create index files.
+
+The example table below is shared by the BTree, Bitmap, Multivalue, Vector, and
+Full-Text guides. It contains three-dimensional sample vectors and is partitioned
+by `dt`, so the partition-scoped build and drop examples also apply. Create and
+populate it once using one of the following alternatives.
<Tabs groupId="global-index-create-table">
-<TabItem value="sql" label="SQL">
+<TabItem value="sql" label="Spark SQL">
```sql
CREATE TABLE my_table (
id INT,
name STRING,
+ category STRING,
+ tag STRING,
tags ARRAY<STRING>,
embedding ARRAY<FLOAT>,
- content STRING
-) TBLPROPERTIES (
+ content STRING,
+ dt STRING
+) PARTITIONED BY (dt) TBLPROPERTIES (
'row-tracking.enabled' = 'true',
- 'data-evolution.enabled' = 'true',
- 'global-index.enabled' = 'true'
+ 'data-evolution.enabled' = 'true'
);
-```
-
-</TabItem>
-<TabItem value="python-sdk" label="Python SDK">
-
-```python
-import pyarrow as pa
-
-from pypaimon import Schema
-
-schema = Schema.from_pyarrow_schema(
- pa.schema([
- pa.field("id", pa.int32()),
- pa.field("name", pa.string()),
- pa.field("tags", pa.list_(pa.string())),
- pa.field("embedding", pa.list_(pa.float32())),
- pa.field("content", pa.string()),
- ]),
- options={
- "row-tracking.enabled": "true",
- "data-evolution.enabled": "true",
- "global-index.enabled": "true",
- },
-)
-
-catalog.create_table("db.my_table", schema, ignore_if_exists=False)
+INSERT INTO my_table VALUES
+ (101, 'a200', 'electronics', 'vip', array('blue', 'green'),
+ array(1.0f, 0.0f, 0.0f), 'paimon lake format and search', '2026-06-18'),
+ (102, 'a300', 'electronics', 'trial', array('green'),
+ array(0.9f, 0.1f, 0.0f), 'paimon full text search', '2026-06-18'),
+ (103, 'a400', 'books', 'standard', array('red'),
+ array(0.0f, 1.0f, 0.0f), 'vector search tutorial', '2026-06-19');
```
</TabItem>
-</Tabs>
-
-## Lifecycle
-
-Create global indexes for all partitions or only selected partitions:
-
-<Tabs groupId="global-index-build">
-
-<TabItem value="sql" label="SQL">
+<TabItem value="flink-sql" label="Flink SQL">
```sql
-CALL sys.create_global_index(
- table => 'db.my_table',
- index_column => 'name',
- index_type => 'btree'
-);
-
-CALL sys.create_global_index(
- table => 'db.my_table',
- index_column => 'name',
- index_type => 'btree',
- partitions => 'dt=2026-06-18;dt=2026-06-19'
-);
-```
-
-</TabItem>
-
-<TabItem value="python-sdk" label="Python SDK">
-
-```python
-table = catalog.get_table("db.my_table")
-
-added_files = table.create_global_index("name")
-print(added_files)
-```
-
-The API returns the number of committed index files. You can pass build options and restrict the
-build to selected partitions:
-
-```python
-added_files = table.create_global_index(
- "name",
- index_type="btree",
- partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
- options={"sorted-index.records-per-range": "10000000"},
-)
-```
-
-</TabItem>
-
-</Tabs>
+SET 'execution.runtime-mode' = 'batch';
-PyPaimon global index build currently supports single-column BTree indexes,
-single-column Bitmap indexes, single-column paimon-vindex vector indexes,
-and single-column full-text indexes on tables with row tracking enabled.
-
-Drop index files:
-
-<Tabs groupId="global-index-drop">
-
-<TabItem value="sql" label="SQL">
-
-```sql
-CALL sys.drop_global_index(
- table => 'db.my_table',
- index_column => 'name',
- index_type => 'btree'
+CREATE TABLE my_table (
+ id INT,
+ name STRING,
+ category STRING,
+ tag STRING,
+ tags ARRAY<STRING>,
+ embedding ARRAY<FLOAT>,
+ content STRING,
+ dt STRING
+) PARTITIONED BY (dt) WITH (
+ 'row-tracking.enabled' = 'true',
+ 'data-evolution.enabled' = 'true'
);
-```
-
-</TabItem>
-<TabItem value="python-sdk" label="Python SDK">
-
-```python
-table = catalog.get_table("db.my_table")
-
-dropped_files = table.drop_global_index("name", index_type="btree")
-print(dropped_files)
-```
-
-You can also restrict the drop to selected partitions, or count matched files without committing:
-
-```python
-matched_files = table.drop_global_index(
- "name",
- index_type="btree",
- partitions=[{"dt": "2026-06-18"}, {"dt": "2026-06-19"}],
- dry_run=True,
-)
+INSERT INTO my_table VALUES
+ (101, 'a200', 'electronics', 'vip', ARRAY['blue', 'green'],
+ ARRAY[CAST(1.0 AS FLOAT), CAST(0.0 AS FLOAT), CAST(0.0 AS FLOAT)],
+ 'paimon lake format and search', '2026-06-18'),
+ (102, 'a300', 'electronics', 'trial', ARRAY['green'],
+ ARRAY[CAST(0.9 AS FLOAT), CAST(0.1 AS FLOAT), CAST(0.0 AS FLOAT)],
+ 'paimon full text search', '2026-06-18'),
+ (103, 'a400', 'books', 'standard', ARRAY['red'],
+ ARRAY[CAST(0.0 AS FLOAT), CAST(1.0 AS FLOAT), CAST(0.0 AS FLOAT)],
+ 'vector search tutorial', '2026-06-19');
```
-</TabItem>
-
-</Tabs>
-
-Global indexes are stored in index files and recorded in table metadata. To inspect index files and
-their row-id coverage, query the `table_indexes` system table:
-
-<Tabs groupId="global-index-table-indexes">
-
-<TabItem value="sql" label="SQL">
-
-```sql
-SELECT index_type, index_field_name, row_range_start, row_range_end
-FROM my_table$table_indexes
-WHERE index_field_name IS NOT NULL;
-```
+Wait for the insert job to finish before building an index.
</TabItem>
<TabItem value="python-sdk" label="Python SDK">
```python
-import pyarrow.compute as pc
-
-table_indexes = catalog.get_table("db.my_table$table_indexes")
-read_builder = table_indexes.new_read_builder().with_projection([
- "index_type",
- "index_field_name",
- "row_range_start",
- "row_range_end",
-])
-
-pa_table = read_builder.new_read().to_arrow(
- read_builder.new_scan().plan().splits()
-)
-pa_table = pa_table.filter(pc.is_valid(pa_table["index_field_name"]))
-print(pa_table)
-```
-
-</TabItem>
-
-</Tabs>
-
-You can also query `file_key_ranges` to inspect data file row-id ranges and diagnose coverage:
-
-<Tabs groupId="global-index-file-key-ranges">
-
-<TabItem value="sql" label="SQL">
-
-```sql
-SELECT file_path, first_row_id, record_count
-FROM my_table$file_key_ranges;
-```
-
-</TabItem>
-
-<TabItem value="python-sdk" label="Python SDK">
+import pyarrow as pa
+from pypaimon import Schema
-```python
-file_key_ranges = catalog.get_table("db.my_table$file_key_ranges")
-read_builder = file_key_ranges.new_read_builder().with_projection([
- "file_path",
- "first_row_id",
- "record_count",
+# `catalog` is the configured Paimon catalog; database `db` already exists.
+pa_schema = pa.schema([
+ ('id', pa.int32()),
+ ('name', pa.string()),
+ ('category', pa.string()),
+ ('tag', pa.string()),
+ ('tags', pa.list_(pa.string())),
+ ('embedding', pa.list_(pa.float32())),
+ ('content', pa.string()),
+ ('dt', pa.string()),
])
-
-pa_table = read_builder.new_read().to_arrow(
- read_builder.new_scan().plan().splits()
+schema = Schema.from_pyarrow_schema(
+ pa_schema,
+ partition_keys=['dt'],
Review Comment:
[P2] Make the shared fixture work with the partition-scoped Python builds
After executing this new setup, the build/rebuild workflow advertised in Manage Global Indexes and the vector guide fails:
```python
table.create_global_index('name', index_type='btree')
table.create_global_index(
'name', index_type='btree',
partitions=[{'dt': '2026-06-18'}, {'dt': '2026-06-19'}],
)
```
The second call raises `IndexError: Position 7 is out of bounds for row arity 1`. `dt` is column 7 in this fixture: `CreateGlobalIndexBuilder._resolve_partition_filter` constructs a full-row predicate, but `build_plan.indexed_row_ranges` applies it directly to the one-field index-manifest partition row. I also reproduced this with the documented IVF-flat calls. Moving `dt` to the first schema position in a control fixture makes the BTree rebuild return `0` as expected. The underlying SDK bug predates this PR, but the new shared walkthrough encounters it. Please provide a working fixture/workaround or explicitly avoid this unsupported sequence until the predicate projection is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [docs] Reorganize multimodal table guides and diagrams [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9721:
URL: https://github.com/apache/paimon/pull/9721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org