[jira] [Updated] (PHOENIX-7998) Vector Indexes

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Kyle Purtell updated PHOENIX-7998:
-----------------------------------------
    Description: 
Vector similarity search uses mathematical embeddings to identify records by semantic meaning and conceptual proximity rather than exact keyword matches. In a hybrid document database like Apache Phoenix, native vector search allows applications to execute unified queries that combine AI-driven semantic retrieval with relational filters and nested BSON document predicates. This eliminates the operational overhead, data replication, and synchronization lag of maintaining external vector databases while leveraging Phoenix's existing scale-out query parallelism.

This design integrates native vector similarity search into Apache Phoenix across relational columns and BSON documents using fixed-dimension vector types, typed path extraction expressions, and standard SQL distance functions. Exact search pushes distance evaluation and bounded candidate heaps down to RegionServers for distributed top-N execution, while approximate retrieval maps Inverted File centroid posting lists into contiguous global index ranges traversed via multi-range skip scans. Server-side coprocessors maintain centroid assignments and verified-write consistency across relational mutations and atomic BSON updates, background compactions observe centroid drift to trigger asynchronous rebuilds, and the optimizer balances exact evaluation, covered index scanning, and adaptive probing within an extensible architecture.

Design Document: https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0

  was:
Vector similarity search, identifying the top _k_ rows whose high dimensional vector embeddings are closest to a query vector, when compared with a given distance metric, is an essential database workload for machine learning embeddings, semantic search, recommendation engines, and retrieval-augmented generation. Modern data platforms across relational, document, and distributed systems provide native vector search capabilities.

This issue proposes native vector indexing and similarity search capabilities for Apache Phoenix. Phoenix on HBase is well structured to support distributed vector search. Horizontal partitioning across regions and distributed top-_k_ aggregation across shards are already implemented through HBase region splits and Phoenix server-side order by, limit, and top-_N_ push down execution.

The implementation introduces a fixed-dimension {{VECTOR}} type utilizing Phoenix's existing packed fixed-width array storage, built-in vector distance functions, and an exact top-_k_ search that executes via the existing server-side top-_N_ scatter-gather engine. This establishes the SQL interface and provides exact nearest-neighbor search and retrieval capabilities. Building on this, it implements a clustering vector index, in which posting lists map directly to contiguous range scans on a covered global index keyed by centroid identifiers. This approach reuses Phoenix's global index mutation pipeline, IndexMaintainer, verified write protocols, read repair mechanisms, and range scan infrastructure. It also implements support for optional per-region immutable HNSW proximity graphs generated during the flush and compaction lifecycles, similar in many respects to Phoenix local indexes, providing high recall at low query latency. Finally, it adds a half-precision data type, scalar quantization, binary quantization, product quantization, and an on-disk graph layout for very high scale deployments.

Design Document: https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0


> Vector Indexes
> --------------
>
>                 Key: PHOENIX-7998
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-7998
>             Project: Phoenix
>          Issue Type: New Feature
>          Components: core
>            Reporter: Andrew Kyle Purtell
>            Priority: Major
>
> Vector similarity search uses mathematical embeddings to identify records by semantic meaning and conceptual proximity rather than exact keyword matches. In a hybrid document database like Apache Phoenix, native vector search allows applications to execute unified queries that combine AI-driven semantic retrieval with relational filters and nested BSON document predicates. This eliminates the operational overhead, data replication, and synchronization lag of maintaining external vector databases while leveraging Phoenix's existing scale-out query parallelism.
> This design integrates native vector similarity search into Apache Phoenix across relational columns and BSON documents using fixed-dimension vector types, typed path extraction expressions, and standard SQL distance functions. Exact search pushes distance evaluation and bounded candidate heaps down to RegionServers for distributed top-N execution, while approximate retrieval maps Inverted File centroid posting lists into contiguous global index ranges traversed via multi-range skip scans. Server-side coprocessors maintain centroid assignments and verified-write consistency across relational mutations and atomic BSON updates, background compactions observe centroid drift to trigger asynchronous rebuilds, and the optimizer balances exact evaluation, covered index scanning, and adaptive probing within an extensible architecture.
> Design Document: https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)