You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2025/11/13 04:05:49 UTC

[PR] Vector index rfc [hudi]

suryaprasanna opened a new pull request, #14255:
URL: https://github.com/apache/hudi/pull/14255

   ### Describe the issue this Pull Request addresses
   
   <!-- Either describe the issue inline here with motivation behind the changes 
        (or) link to an issue by including `Closes #<issue-number>` for context. 
        If this PR includes changes to the storage format, public APIs,
        or has breaking changes, use `!` (e.g., feat!: ...) -->
   
   ### Summary and Changelog
   
   <!-- Short, plain-English summary of what users gain or what changed in behavior.
        Followed by a detailed log of all the changes. Highlight if any code was copied. -->
   
   ### Impact
   
   <!-- Describe any public API or user-facing feature change or any performance impact. -->
   
   ### Risk Level
   
   <!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
        If medium or high, explain what verification was done to mitigate the risks. -->
   
   ### Documentation Update
   
   <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
   
   - The config description must be updated if new configs are added or the default value of the configs are changed.
   - Any new feature or user-facing change requires updating the Hudi website. Please follow the 
     [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2525709157


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.

Review Comment:
   how do we organize these clusters, hashing by keys? can one cluster includes multiple file groups here from Hudi's notion?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3528314248

   Thanks @suryaprasanna for making this!
   
   I was wondering if you can create a github issue for your RFC similar to something like this https://github.com/apache/hudi/issues/14219 and then link your RFC there as well. Also wondering if can change the title for both the issue and this PR to be something like `RFC-103 - Vector Index in Hudi`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3537775173

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f42e82404e2d43df0c37bd4d881d79e25e2d9f1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6f42e82404e2d43df0c37bd4d881d79e25e2d9f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   * 4f01389ab8b48eb1164e856e5296beb6424eb75d UNKNOWN
   * 3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7 UNKNOWN
   * a3d331882497784cfc2681a67e778f113bee2fa2 UNKNOWN
   * 6f42e82404e2d43df0c37bd4d881d79e25e2d9f1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2523899254


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.

Review Comment:
   I am interested in this idea, you might cover later in the RFC around what our strategies are for updating the index again since I am assuming this is expensive. If we are do on every commit which we pay the cost on each write, or if we offer follow hudi current model of having this trigger x commits or some time interval to reindex?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)

Review Comment:
   Will link the draft RFC 102: https://github.com/apache/hudi/issues/14219 around exposing a SQL interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-4185163522

   > RFC-103 is already taken. We need to first grab a new RFC number for this.
   
   Sure Vinoth, will take the latest RFC that is available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3543513994

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9576",
       "triggerID" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 945a8ffa92f6ebd5a4050d5d2cb2641f079a3161 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9576) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2523983482


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.

Review Comment:
   Should we track any other additional metadata such as what was the embedding model used to generate this column, or what is the blob/binary col this embedding column maps to?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r3034715863


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.

Review Comment:
   🤖 The requirements list multi-index support, upserts, deletes, rollback, and refresh — but doesn't mention schema evolution. What happens if the embedding dimension changes (e.g., migrating from a 768-dim to a 1536-dim model)? This seems like it would require a full index rebuild. Worth calling out explicitly.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.

Review Comment:
   🤖 The RFC does touch on this further down — incremental writes land in log files during ingestion (lightweight, no index rebuild), and the actual vector index under `.index_files` is only rebuilt during compaction. So it follows Hudi's existing model of deferring expensive work to a table service. That said, it would be worth calling this out more explicitly up front, since the cost profile of rebuilding an ANN graph is quite different from, say, compacting columnar files — the rebuild time scales with the total index size, not just the delta.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.
+
+##### Notes on File Groups and Clusters
+
+There is no one-to-one mapping between a vector index’s file groups and its clusters. For example, a vector index may use five file groups to distribute log-file writes evenly, while the underlying index structure may contain only two clusters.
+
+- **Number of file groups** is influenced by factors such as base file size, ingestion parallelism, and writer throughput.
+- **Number of clusters** is determined by ANN read performance requirements, index storage layout, and other vector index configuration settings.
+
+Exactly how the compaction operation happens will be discussed in the Refresh index Workflow section.
+
+#### Refresh Index / Compaction workflow:
+
+Ingestion workflow stores the changes to vector embeddings in a vector_index partition directory in log file format. These are not readily available for user to read. For users to consume these changes they need to be applied on the vector indexes that are present under .index_files directory. Hudi’s Compaction is the table service that applies these changes on to the index files.
+
+Compaction for the vector index partition is significantly more complex than compaction for other metadata partitions. This complexity arises because vector index files are stored outside the traditional base-file and log-file structure, and each new version of a vector index is maintained independently from its previous version. Since vector indexes do not support transaction-based change tracking and cannot roll back partial updates, version management must be implemented externally. This external versioning ensures that the currently serving index remains isolated and stable while a new version is being built and updated during compaction.
+
+<img src="./rfc-103-vector_index_compaction.jpeg"/>
+
+The first step in compaction is to create a new version directory for the column being compacted. This directory is located under:
+```text
+vector_index/.index_files/column=<column_name>/version=<x>
+```
+where **<x>** is the next version number.
+
+Once the version directory is created, the system begins merging log records. These records are split into upserts and deletes:
+- **Deletes:** Records that have been removed from the main table.
+- **Upserts:** Records that are either new inserts or updates to existing vectors.
+
+Because updates to existing vectors are more complex—only the latest entries are known during metadata writes—the system uses the base files to differentiate between inserts and updates. Updates are further broken down into:
+- **Deletes:** Remove the older vector entries from the index.
+- **Inserts:** Add the updated vector values as new entries.
+
+Both types of deletes (actual deletions and updates) are combined and applied in bulk across all relevant clusters.
+
+First deletions are processed, then the system handles the inserts. Inserts are processed in two batches:
+- **New vectors:** Records added for the first time.
+- **Updated vectors:** Records that were part of updates.
+
+These insert batches are combined, run through the clustering algorithm, and then inserted into the appropriate vector index clusters. After all inserts are complete, column-specific base files are created to reflect the updated vector entries. Once the base files are finalized, the metadata files for the column are updated to point to the new version of the index.
+
+These metadata files track:
+- The location of the base files containing the vector embeddings
+- The version reference corresponding to the compaction commit
+- Any other information related to the vector embeddings or index
+
+When the compaction commit completes, all vector searches are automatically routed to the updated index in the new version directory, ensuring consistency and correctness.
+
+**Note:**
+- When vector index is configured for multiple columns then compaction/refresh index for each column can be done either together or separately.
+- Since, compaction plan is a immutable plan any failure have to be retried, and during reattempt, all the in-progress version directories will be deleted and compaction process is started. Example, let us a column’s vector_index has version of 3, that means version=3 is the directory serving the request, so the new compaction operation will create version=4 and work on it. If this compaction operation fails then it will delete the version=4 directory and restart compaction operation again with the same plan.
+
+## Rollout/Adoption Plan
+
+## Test Plan

Review Comment:
   🤖 There's no 'Alternatives Considered' section. It would strengthen the RFC to discuss why storing vector indexes inside the metadata table was chosen over alternatives like: (1) a sidecar index file per file group (similar to bloom filters), (2) leveraging an external vector index service, or (3) storing indexes as a separate Hudi table rather than a metadata partition.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector

Review Comment:
   🤖 The column ID mapping is described as permanent even after the index is dropped. What's the garbage collection story for orphaned vector index data? If a user drops a vector index, are the corresponding files under vector_index/ cleaned up immediately, or do they linger until a full rebootstrap?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2523901924


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)

Review Comment:
   @suryaprasanna I am thinking maybe for these two bullet points, we move these items scope of RFC 102, since it handles exposing the capability to perform to the user.
   ```
   - First-class SQL experience for creating, and dropping indexes (Spark)
   - SQL extensions to query the index. (Spark, then Presto/Trino)
   ```
   Will link the draft RFC 102: https://github.com/apache/hudi/issues/14219 around exposing vector search 
   
   I also think that we have SQL semantics already for creating and dropping indexes in spark, for secondary index https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSecondaryIndexDataTypes.scala#L190C1-L191C1. So wondering if we can leverage similar wiring?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2530273290


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)

Review Comment:
   I guess for these RAG applcations, can the rag application just be a python script shared via pyspark? The developer can then directly invoke our vector search api from RFC 102? I saw this basic blog on RAG https://huggingface.co/learn/cookbook/en/rag_with_hf_and_milvus



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3537726474

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   * 4f01389ab8b48eb1164e856e5296beb6424eb75d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r3034716312


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.

Review Comment:
   🤖 Agreed — the text at line 108 should be reworded. The proposed algorithms section above already lists graph-based (HNSW), JVM-native (jVector), and C++ library (FAISS which supports IVF/PQ/flat), so the characterization of ANN structures as "graph-based" in this paragraph contradicts the RFC's own earlier section.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status

Review Comment:
   🤖 The Status field is blank. Could you fill this in (e.g., UNDER REVIEW) so readers know where the RFC stands?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+

Review Comment:
   🤖 The compaction-based refresh means vector search results can be arbitrarily stale between compaction runs. Could you discuss the expected staleness window and whether there's a way for users to configure or bound it? For example, could the read path optionally merge log files at query time (like MOR snapshot reads) to reduce staleness, even at the cost of performance?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.
+
+##### Notes on File Groups and Clusters
+
+There is no one-to-one mapping between a vector index’s file groups and its clusters. For example, a vector index may use five file groups to distribute log-file writes evenly, while the underlying index structure may contain only two clusters.
+
+- **Number of file groups** is influenced by factors such as base file size, ingestion parallelism, and writer throughput.
+- **Number of clusters** is determined by ANN read performance requirements, index storage layout, and other vector index configuration settings.
+
+Exactly how the compaction operation happens will be discussed in the Refresh index Workflow section.
+
+#### Refresh Index / Compaction workflow:
+
+Ingestion workflow stores the changes to vector embeddings in a vector_index partition directory in log file format. These are not readily available for user to read. For users to consume these changes they need to be applied on the vector indexes that are present under .index_files directory. Hudi’s Compaction is the table service that applies these changes on to the index files.
+
+Compaction for the vector index partition is significantly more complex than compaction for other metadata partitions. This complexity arises because vector index files are stored outside the traditional base-file and log-file structure, and each new version of a vector index is maintained independently from its previous version. Since vector indexes do not support transaction-based change tracking and cannot roll back partial updates, version management must be implemented externally. This external versioning ensures that the currently serving index remains isolated and stable while a new version is being built and updated during compaction.
+
+<img src="./rfc-103-vector_index_compaction.jpeg"/>
+
+The first step in compaction is to create a new version directory for the column being compacted. This directory is located under:
+```text
+vector_index/.index_files/column=<column_name>/version=<x>
+```
+where **<x>** is the next version number.
+
+Once the version directory is created, the system begins merging log records. These records are split into upserts and deletes:
+- **Deletes:** Records that have been removed from the main table.
+- **Upserts:** Records that are either new inserts or updates to existing vectors.
+
+Because updates to existing vectors are more complex—only the latest entries are known during metadata writes—the system uses the base files to differentiate between inserts and updates. Updates are further broken down into:
+- **Deletes:** Remove the older vector entries from the index.
+- **Inserts:** Add the updated vector values as new entries.
+
+Both types of deletes (actual deletions and updates) are combined and applied in bulk across all relevant clusters.
+
+First deletions are processed, then the system handles the inserts. Inserts are processed in two batches:
+- **New vectors:** Records added for the first time.
+- **Updated vectors:** Records that were part of updates.
+
+These insert batches are combined, run through the clustering algorithm, and then inserted into the appropriate vector index clusters. After all inserts are complete, column-specific base files are created to reflect the updated vector entries. Once the base files are finalized, the metadata files for the column are updated to point to the new version of the index.
+

Review Comment:
   🤖 The Rollout/Adoption Plan and Test Plan sections are empty. For an RFC of this scope, could you outline at least: (1) the phasing of delivery (which workflow ships first), (2) whether this is gated behind a feature flag, (3) backward compatibility guarantees (what happens when a reader without vector index support encounters a metadata table with a vector_index partition), and (4) key test scenarios (bootstrap correctness, concurrent compaction, rollback after partial write)?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.

Review Comment:
   🤖 The clustering algorithm is central to this design but underspecified. How are initial cluster boundaries determined — k-means, random partitioning, something else? More importantly, how do clusters evolve as data distribution shifts over time? If cluster boundaries are fixed at bootstrap, index quality could degrade significantly as new vectors drift away from original centroids.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.

Review Comment:
   🤖 Good point. At minimum, tracking the embedding model name and dimensionality in the index metadata would let the system validate at query time that the query vector is compatible with the indexed embeddings. Without this, a model swap upstream could silently produce nonsensical similarity results.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.
+
+##### Notes on File Groups and Clusters
+
+There is no one-to-one mapping between a vector index’s file groups and its clusters. For example, a vector index may use five file groups to distribute log-file writes evenly, while the underlying index structure may contain only two clusters.
+
+- **Number of file groups** is influenced by factors such as base file size, ingestion parallelism, and writer throughput.
+- **Number of clusters** is determined by ANN read performance requirements, index storage layout, and other vector index configuration settings.
+
+Exactly how the compaction operation happens will be discussed in the Refresh index Workflow section.
+
+#### Refresh Index / Compaction workflow:
+
+Ingestion workflow stores the changes to vector embeddings in a vector_index partition directory in log file format. These are not readily available for user to read. For users to consume these changes they need to be applied on the vector indexes that are present under .index_files directory. Hudi’s Compaction is the table service that applies these changes on to the index files.
+
+Compaction for the vector index partition is significantly more complex than compaction for other metadata partitions. This complexity arises because vector index files are stored outside the traditional base-file and log-file structure, and each new version of a vector index is maintained independently from its previous version. Since vector indexes do not support transaction-based change tracking and cannot roll back partial updates, version management must be implemented externally. This external versioning ensures that the currently serving index remains isolated and stable while a new version is being built and updated during compaction.
+
+<img src="./rfc-103-vector_index_compaction.jpeg"/>
+
+The first step in compaction is to create a new version directory for the column being compacted. This directory is located under:
+```text
+vector_index/.index_files/column=<column_name>/version=<x>
+```

Review Comment:
   🤖 The rollback/restore strategy needs more detail. You mention version directories and that a failed compaction deletes the in-progress version, but what happens when a commit on the main table is rolled back? How are the corresponding log file entries in the vector_index partition invalidated? Is there a risk of ghost vectors remaining in the index after a rollback?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.

Review Comment:
   🤖 For the read flow, collecting top-K from each cluster on the driver could be a bottleneck for large K or many clusters. Have you considered a distributed merge step instead of collecting to the driver? Also, is there a plan for supporting distance thresholds (range queries) in addition to top-K?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.

Review Comment:
   🤖 IVF could actually pair well with the cluster-based approach proposed here — IVF's coarse quantization partitions map naturally to Hudi's cluster concept, and the inverted lists are simpler to merge incrementally than graph structures like HNSW. Might be worth considering as a second algorithm option alongside HNSW/jVector, especially for very large-scale indexes where rebuild cost matters.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.

Review Comment:
   🤖 It might be worth discussing the trade-offs between HNSW, jVector, and FAISS in more structured way — memory footprint, JNI overhead, license compatibility (FAISS is MIT but uses native code), and whether the design allows swapping implementations later. A pluggable `VectorIndexEngine` interface would make the choice less committal.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2530273290


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)

Review Comment:
   @suryaprasanna For these RAG applcations, can the rag application just be a python script shared via pyspark? The developer can then directly invoke our vector search api from RFC 102? I saw this basic blog on RAG https://huggingface.co/learn/cookbook/en/rag_with_hf_and_milvus that shows an example with a vector database like milvus, so wondering if instead you swap out milvus with hudi acting as the vector store, and using pyspark as execution layer.
   
   Or is spark not suited for these kinda use cases?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2523901924


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)

Review Comment:
   Will link the draft RFC 102: https://github.com/apache/hudi/issues/14219 around exposing vector search 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2544289354


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector

Review Comment:
   can we produce some microbenchmarks for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3543503095

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 945a8ffa92f6ebd5a4050d5d2cb2641f079a3161 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed pull request #14255: docs: RFC-103 - Support Vector Indexing in Apache Hudi
URL: https://github.com/apache/hudi/pull/14255


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3555414230

   @rahil-c @suryaprasanna can we merge this and the other RFC PR?
   
   Also please follow process and grab a RFC number


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2525716618


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+#### Metadata directory structure:
+
+![Vector index metadata structure](rfc-103-metadata_structure.jpeg)
+
+
+#### Vector Index partition directory structure:
+
+![Vector index partition structure](rfc-103-vector_index_hudi_files.jpeg)
+
+#### Index files directory structure:
+
+![Vector index index files](rfc-103-vector_index_index_files_file_structure.jpeg)
+
+#### Base & Log Files:
+
+In Hudi’s vector index implementation actually embeddings are inserted under the index files, base files contain metadata information about the index. File groups used for creating base files will be of the format **<COLUMN_NAME>-<FILE_INDEX>_<WRITE_TOKEN>_<COMMIT>.hfile**. Here, column_name is the column for which the vector index is created. This way when there are multiple vector indices part of the same dataset, compaction operation can be performed on each vector index separately. There by avoiding large scale updates to all the indexes at the same time.

Review Comment:
   The base file name in the example graph does not really follow this naming convention: `**<COLUMN_NAME>-<FILE_INDEX>_<WRITE_TOKEN>_<COMMIT>.hfile**`



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.

Review Comment:
   And one base file could have multiple index columns right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2526009591


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.

Review Comment:
   If i understand correctly these clusters are basically rows that are grouped together by how similar their embedding values are to one another, and not directly related to the recordkey being hashed unless you are talking about some other value being used as key? @danny0405  @suryaprasanna 
   
   https://www.pinecone.io/learn/a-developers-guide-to-ann-algorithms/
   
   ```
   Based on how vectors are structured, vector indexing algorithms can be divided into three main categories:
   
   Spatial Partitioning (also known as cluster-based indexing or clustering indexes)
   Graph-based indexing (e.g. [HNSW](https://www.pinecone.io/learn/series/faiss/hnsw/))
   Hash-based indexing (e.g., [locality-sensitive hashing](https://www.pinecone.io/learn/series/faiss/vector-indexes/#Locality-Sensitive-Hashing))
   We’ll skip hash-based indexing in this article because its performance on all aspects, including reads, writes, and storage, is currently worse than that of graph-based and spatial partitioning-based indexing. Almost no vector databases use hash-based indexing nowadays.
   
   
   Inverted File Index (IVF)
   Another popular index is the Inverted File Index (IVF). It’s easy to use, has high search quality, and has reasonable search speed.
   
   It’s the most basic version of the spatial-partitioning index and inherits its advantages including little space overhead and the ability to work with object storage as well as disadvantages like lower query throughput compared to graph-based indexing in-memory and on-disk.
   ```
   
   When I was reading online I saw that IVF seemed to be popular with object storage and seems to be a default in other systems in the space https://lancedb.com/docs/indexing/#:~:text=Inverted%20File%20Index%20(IVF)%20Implementation,but%20the%20slower%20the%20query.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3537789266

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6f42e82404e2d43df0c37bd4d881d79e25e2d9f1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6f42e82404e2d43df0c37bd4d881d79e25e2d9f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   * 4f01389ab8b48eb1164e856e5296beb6424eb75d UNKNOWN
   * 3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7 UNKNOWN
   * a3d331882497784cfc2681a67e778f113bee2fa2 UNKNOWN
   * 6f42e82404e2d43df0c37bd4d881d79e25e2d9f1 UNKNOWN
   * 945a8ffa92f6ebd5a4050d5d2cb2641f079a3161 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3525218870

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2523920185


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.

Review Comment:
   @suryaprasanna I am curious if you looked into IVF and if its actually a good candidate to be implemented for hudi's internal metadata table. https://blog.dailydoseofds.com/p/approximate-nearest-neighbor-search



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2525700739


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.

Review Comment:
   I'm wondering if the Locality-Sensitive Hashing(LSH) is a good candicate for hashing index, because of it's efficiency and may be more friendly for streaming ingestion? https://www.cs.princeton.edu/courses/archive/spring13/cos598C/Gionis.pdf



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-4174146475

   RFC-103 is already taken. We need to first grab a new RFC number for this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3537765126

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a3d331882497784cfc2681a67e778f113bee2fa2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   * 4f01389ab8b48eb1164e856e5296beb6424eb75d UNKNOWN
   * 3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7 UNKNOWN
   * a3d331882497784cfc2681a67e778f113bee2fa2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2544183544


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.

Review Comment:
   so would this build the data structure needed for ANN each time a query is executed? 



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status

Review Comment:
   ??



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.

Review Comment:
   we should fit this into the standard hudi file naming. 



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.
+
+##### Notes on File Groups and Clusters
+
+There is no one-to-one mapping between a vector index’s file groups and its clusters. For example, a vector index may use five file groups to distribute log-file writes evenly, while the underlying index structure may contain only two clusters.

Review Comment:
   Interesting.. is it 1:N? can we clarify.. 



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.

Review Comment:
   each column should be its own vector index?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.

Review Comment:
   would nt updates cause changes to cluster boundaries.. should this be readjusted based on periodic Hudi clustering..



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.

Review Comment:
   vector search index is a secondary index IMO



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.

Review Comment:
   Should n't we just shuffle the first stage and compute the final top K .. distributed fashion?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.
+
+For example, if a column’s embeddings are split into two clusters, the system generates two separate vector indexes for that column.
+
+This process is repeated for every column where vector indexing is enabled. Thus, if two columns each produce two clusters, a total of four vector indexes are constructed.
+
+#### Incremental workflow:
+
+After data is written into the main table, the ingestion flow for the vector_index partition begins. The first step is to split incoming records based on the vector index configurations and the file groups associated with each index. This splitting is required whenever multiple vector indexes are defined.
+
+<img src="./rfc-103-vector_index_ingestion_and_compaction_overview.jpeg"/>
+
+For example, consider a record `(r1, v1, v2)` where `r1` is the `record_key`, and `v1` and `v2` are embeddings for two different vector indexes. This record is split into two independent records: `(r1, v1)` and `(r1, v2)`. Each split record is then routed to the file groups corresponding to its specific vector index—`(r1, v1)` goes to the file groups for the `v1` index, while `(r1, v2)` goes to those for the `v2` index.
+
+Once the split is complete, the records destined for a specific vector index are evenly distributed across the file groups configured for that index and written into their log files. Importantly, the vector index base files stored under `vector_index/.index_files` are not updated during ingestion. Therefore, SQL queries that read only the index files will not immediately reflect changes written to log files. The .index_files are updated only during compaction. This separation ensures ingestion remains lightweight and avoids expensive index rebuild operations.
+
+This design intentionally stores records for each vector index under column-specific file groups. The purpose is to avoid conflicts when multiple compaction plans run concurrently—each targeting different vector index columns. Because of this structure, when a compaction operation is triggered to refresh a specific vector index, only the file groups belonging to that index are scanned. The system does not need to scan every file group in the vector_index partition, enabling more efficient and isolated compaction.
+
+##### Notes on File Groups and Clusters
+
+There is no one-to-one mapping between a vector index’s file groups and its clusters. For example, a vector index may use five file groups to distribute log-file writes evenly, while the underlying index structure may contain only two clusters.
+
+- **Number of file groups** is influenced by factors such as base file size, ingestion parallelism, and writer throughput.
+- **Number of clusters** is determined by ANN read performance requirements, index storage layout, and other vector index configuration settings.
+
+Exactly how the compaction operation happens will be discussed in the Refresh index Workflow section.
+
+#### Refresh Index / Compaction workflow:
+
+Ingestion workflow stores the changes to vector embeddings in a vector_index partition directory in log file format. These are not readily available for user to read. For users to consume these changes they need to be applied on the vector indexes that are present under .index_files directory. Hudi’s Compaction is the table service that applies these changes on to the index files.
+
+Compaction for the vector index partition is significantly more complex than compaction for other metadata partitions. This complexity arises because vector index files are stored outside the traditional base-file and log-file structure, and each new version of a vector index is maintained independently from its previous version. Since vector indexes do not support transaction-based change tracking and cannot roll back partial updates, version management must be implemented externally. This external versioning ensures that the currently serving index remains isolated and stable while a new version is being built and updated during compaction.

Review Comment:
   I want to fit this all under the current storage format/model



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:

Review Comment:
   lets call this `indexer` or `index building` consistent with other SI building workflows..



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:
+
+Since vector indexes are stored within Hudi’s metadata table, they are created as part of the standard metadata initialization workflow. During this workflow, metadata partitions are generated iteratively—each new partition can reuse information generated by earlier ones. When the vector index partition is initialized, it uses the file-level metadata to locate and read all data files that contain embedding vectors.
+
+<img src="./rfc-103-vector_index_bootstrap_process.jpeg"/>
+
+##### Cluster Formation and Index Construction
+
+After loading the embedding vectors from the data files, the system groups them into clusters. Clustering provides horizontal scalability for vector search, preventing the bottlenecks associated with a single monolithic index. Each cluster is processed independently using the configured indexing algorithm, resulting in one vector index per cluster.

Review Comment:
   Can we add details on how exactly this clustering algorithm is done. are we envisioning use of sth like k-means 



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)

Review Comment:
   why spark specifically.. 



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping

Review Comment:
   again. we should separate the storage changes out and fit on top of existing infra/format.



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.

Review Comment:
   okay again. we have a lot of this already for the SI support.. lets reuse all that..



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.

Review Comment:
   My understanding is : we are grouping the embeddings by some clustering algo. and eventually, each cluster group maps to a file group in the vector search index at the Hudi table level?



##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,267 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-103: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.
+
+However, in large-scale batch processing environments such as Apache Spark or Apache Flink, horizontal scaling—distributing computation across multiple nodes—is the standard and more efficient approach for handling data at scale.
+
+For example, when performing a vector-based join between two large datasets using nearest neighbor similarity, the operation must ultimately execute in a distributed framework. To align with this architecture and industry practices, it is essential to design a mechanism that enables horizontal scalability of vector indexes across multiple nodes.
+
+##### Proposed Approach
+
+To enable horizontal scalability, we propose organizing the vector embeddings into multiple clusters of indexes. Each cluster will manage a subset of the embeddings, allowing the overall dataset to be evenly partitioned across the clusters.
+During execution, each Spark executor can load the relevant cluster’s index into memory—or operate in a hybrid mode using both disk and memory—to perform local nearest neighbor searches efficiently.
+
+#### Read Flow on Vector Index:
+
+When users are accessing the vector index on a column embeddings to find top K nearest neighbours for a vector. The query will first start a spark stage to find the top K nearest neighbours across each of the clusters and the work will be distributed using Spark. Let us say there are 10 clusters, then total 10xK vectors are collected as intermediary result and as part of final step from these 10xK vectors, another top K vectors that are nearest is computed based by on the driver and returned.
+
+
+#### Supporting transactions for Upsert/deletes:
+Vector embeddings are not static—they can evolve over time as underlying features change or as new embedding generation algorithms are applied. Therefore, the vector index implementation must support upserts and deletes, along with integration into Hudi table services such as rollback and restore operations.
+
+To achieve this, we propose using a versioning mechanism for maintaining and updating the vector index. Each update to the embeddings or index structure can be associated with a version, enabling consistent rollback and recovery when table operations require reverting to a previous state.
+
+In real-time systems such as Pinecone or similar vector databases, upserts and deletes are handled instantly, and read queries reflect these updates almost immediately due to their in-memory index design. However, in batch processing environments, vector indexes often rely on disk-based graph structures. In this context, it is more efficient to collect updates over time and apply them in bulk during a scheduled index refresh. This refresh process can be seamlessly aligned with Hudi’s compaction operation, ensuring that the vector index remains synchronized with the underlying dataset while maintaining efficient batch performance.
+
+### Data Layout:
+
+The vector index data will be stored under the metadata table’s vector_index partition. Within this partition, there will be mainly three types of files/directories:
+1. **Vector embedding files:** These files will store the actual vector embeddings along with their associated record keys. They will be organized by column and cluster to facilitate efficient access and updates. Both base and log files will be used to manage the vector embeddings. These files will either be of the form **vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>** for base files or **.vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<COMMIT_TIME>_<WRITE_TOKEN>** for log files.
+2. **Metadata tracking base files:** These files will maintain metadata about the vector indexes, including version information, index locations, and other relevant details necessary for managing the index lifecycle.
+3. **.index_files directory:** This directory will contain the vector-index.properties file that maps column names to unique column IDs for safe and deterministic file naming. This directories also contain column specific versioned directories to manage different iterations of the index, allowing for rollback and recovery.
+
+<img src="./rfc-103-vector_index_data_layout.jpeg"/>
+
+#### Base File Naming Scheme
+
+There is a possibility of column names containing characters like '_' or '-', which complicate filename parsing and create edge cases.
+
+To avoid these issues, the design adopts the naming style already used in other metadata partitions (e.g., files-0001, record-index-0001). Vector index base files will follow the pattern as shown below:
+
+```
+vectors-<COLUMN_ID>-<FILE_GROUP>-<FILE_PREFIX>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+#### Column Name to Column ID Mapping
+
+If column_id are used as part of the file names, then there needs to be a mapping maintained between column names and column ids. This mapping can be stored as part of a properties file.
+
+A static properties file is required to store the mapping between column names and their assigned column IDs. This file resides under the .index_files directory and it can be named as **vector-index.properties**.
+
+This file is written only during vector index creation, which is part of a metadata initialization code block, which is under a table lock. This ensures there are no race conditions. Also, once a column receives a column ID, the mapping is permanent. That means even if the vector is disabled the mapping cannot be removed and the same column ID must be reused if the vector index is re-enabled in the future. Only entire vector index reboostrap removes this mapping and recreates it from scratch.
+
+Example contents of vector-index.properties:
+
+```text
+vector.index.columns=product_vector,category_vector
+vector.index.product_vector=000
+vector.index.category_vector=001
+```
+
+This allows the system to deterministically and safely resolve column names to their internal identifiers during all future ingestion, compaction, and indexing operations.
+
+#### Base Files for Vector Index Tracking
+
+Once indexing is complete, the mapping between record_key and embedding vectors is stored in the base files within the metadata table. This mapping enables the system to correctly handle updates and deletions—old vectors can be located and removed, and new embeddings can be inserted during compaction or incremental operations.
+
+In addition to the base files containing embeddings, each vector index requires a dedicated base file to track its version, location, and associated metadata. These files are essential for switching between old and new versions of a vector index, since the rest of the system relies on compaction commit completion to determine which version is active.
+
+Therefore, for every column with a configured vector index, a corresponding tracking base file is created. For example, if two columns have vector indexes, two tracking base files will be generated. These files will follow the naming convention outlined below.
+
+```text
+vector-index-metadata-<COLUMN_ID>_<WRITE_TOKEN>_<COMMIT_TIME>
+```
+
+### Workflows:
+There are various workflows involved in managing vector indexes as part of Hudi table services. Following are the main workflows that needs to be implemented as part of this RFC.
+
+- Bootstrap workflow
+- Incremental workflow
+- Refresh Index / Compaction workflow
+
+Expanding on these workflows, following sections will go over the data layout and directory structure of vector index implementation.
+
+#### Bootstrap workflow:

Review Comment:
   all of these parameters should be fed nicely into a `CREATE INDEX ..` command syntax



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3537756062

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a71a6f67d5416e13453104d6d0e2bcdadf241945",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f01389ab8b48eb1164e856e5296beb6424eb75d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a71a6f67d5416e13453104d6d0e2bcdadf241945 UNKNOWN
   * 4f01389ab8b48eb1164e856e5296beb6424eb75d UNKNOWN
   * 3ff0fd9ebb63e03f8ed20b4929a17f4668941ab7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2525706917


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.

Review Comment:
   > are graph-based structures
   
   The main families of ANN indexing structures are graph-based, tree-based, hashing-based, and quantization-based methods, not just graph-based?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Vector index rfc [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #14255:
URL: https://github.com/apache/hudi/pull/14255#discussion_r2525963595


##########
rfc/rfc-103/rfc-103.md:
##########
@@ -0,0 +1,178 @@
+   <!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+# RFC-101: Support Vector Index on Hudi
+
+## Proposers
+
+- @suryaprasanna
+- @prashantwason
+
+## Approvers
+- @vinoth
+- @rahil-c
+
+## Status
+
+
+## Abstract
+As LLM applications are on the rise, a lot of focus has been on “operational” or “online” databases, that have added vector search capabilities or specialized vector databases (Chroma, Pinecone, ..), which offer similar capabilities. Specialized vector databases claim support for better algorithms, optimized ingest/serving performance and better integration with LLM application development frameworks like Langchain/llamaIndex et al.
+
+With its shared-storage/decoupled compute model, the data lakehouse architecture has already proven scalability and cost-effectiveness advantages compared to storing all data for analysis and processing in shared-nothing database architectures or datawarehouses. We believe that extending data lakehouse storage and query engines with vector search capabilities can unlock best-of-both-worlds with some exciting outcomes.
+
+Storing vector indexes in a data lakehouse offers several advantages:
+- **Infinitely scalable storage:** 
+  - Overcomes the pain of storing/scaling large amounts of embeddings in an online database forever, reducing costs, while also allowing the production database to be more efficient/easier to operate.
+- **Leverage scalable compute frameworks:** 
+  - There is already rich support for compute frameworks (Spark, Flink) to build ingest pipelines to maintain embeddings from upstream sources, as well as fast query engines (Presto, Trino, Starrocks) that can serve vector searches.
+- **Tiered serving layer:** 
+  - Given much of the embedding data is updated from data pipelines, often every so often (versus real-time individual updates),  we can also provide the reasonably fast serving of vector queries (e.g. 80% speed at 80% lower cost) by either extending the lakehouse storage with a caching tier (or) a tiered storage integration into existing production/online vector databases. They could serve applications with different end-user expectations e.g internal business apps vs user-facing applications.
+
+By extending the multi-modal indexing subsystem, vector indexes can be stored as part of the Hudi’s metadata tables and can be served directly using
+
+
+## Background
+Following are the goals for this RFC
+- Creating vector indexes based on a base column in a table - either an embedding column or a text column.
+- Indexes are automatically kept up to date when the base column changes, consistent with transactional boundaries.
+- First-class SQL experience for creating, and dropping indexes (Spark)
+- SQL extensions to query the index. (Spark, then Presto/Trino)
+
+Non-goals/Unclear:
+- Fast serving layer, directly usable from RAG applications (this can be left to existing ODBC/SQL gateways that can talk to Spark?)
+- Integration within the ecosystem - langchain , llamaIndex. … and developer experience.
+
+Hudi, with its extensible multi-modal indexing subsystem, already offers a lot of capabilities for us to build this e2e architecture. Hudi already supports - column statistics, files, bloom filters, and record-level - indexes in 0.x release line, while 1.x already has  - functional indexes, secondary indexing, text indexes - with integrations into Spark SQL.
+
+Specifically, the indexing infrastructure provides the following capabilities
+- Async index creation, allowing for indexes to be (re)built without impacting the writers of the table.
+- Allowing queries to gracefully degrade, by an index discovery mechanism.
+- Concurrency control mechanisms to keep index and data integrity in-tact, even in the face of failures.
+- Hudi table services automatically manage index data.
+- Extensible to allow new index types to be integrated easily.
+
+The overall idea is to see if we can add a new vector search index, into the Hudi indexing subsystem. We can directly leverage Hudi’s strengths around table management, ingest tooling and diverse writers - while doing some interesting extensions to turn Hudi into a direct serving layer.
+
+## Implementation
+
+### Approximate Nearest Neighbor (ANN) for Vector Indexing
+To efficiently find the nearest neighbors of a vector, the industry-standard approach is to use Approximate Nearest Neighbor (ANN) algorithms.
+
+While exact search methods (e.g., direct cosine distance computation) achieve 100% accuracy, they are computationally expensive and scale poorly for large datasets. In contrast, ANN indexing structures typically provide around 95% accuracy while offering 10× to 100× faster retrieval times.
+
+Given these trade-offs, ANN-based indexing is the most practical and performant choice for implementing vector search in Apache Hudi. Recent advances in AI and vector search technologies have led to several efficient ANN algorithms. Depending on the use case and environment, the following are recommended options:
+
+1. HNSW (Hierarchical Navigable Small World) – implemented via hnswlib-ann
+   - Graph-based approach
+   - Excellent recall-speed tradeoff
+   - Widely used in production systems
+2. jVector
+   - Java-native implementation of ANN structures
+   - Suitable for JVM-based environments like Apache Hudi
+3. FAISS (Facebook AI Similarity Search)
+   - Highly optimized C++ library with Python bindings
+   - Ideal for large-scale GPU-accelerated similarity search
+
+For initial implementation, we will look into HNSW and Jvector implementations.
+
+### Requirements:
+Before going over the architecture of Vector index first let us go through the basic requirements,
+
+- **Multi-index support:** The implementation must support maintaining multiple vector indexes, each corresponding to different columns.
+- **Data mutation handling:** The vector index must support upserts, inserts, and deletes to stay consistent with the changes happening on the underlying dataset.
+- **Transactional consistency:** The implementation must support rollback all vector index changes when commits are reverted or removed during restore or rollback operations.
+- **Index refresh capability:** The implementation must provide an ability to refresh or rebuild vector indexes after data or configuration changes to ensure accuracy and consistency.
+
+### Vector Index stored under Metadata table:
+Each Hudi dataset contains a Metadata table, which is stored under the .hoodie directory of the dataset. This Hudi Metadata table is another Hudi table which uses MOR table type. This internal table is used for storing Files metadata, Column Stats, Record indexes and Secondary Indexes etc related to the main Hudi dataset. Hudi Metadata table is not a centralized storage for all Hudi datasets, it is tightly coupled with the main dataset and every commit on the main table contain a corresponding commit in the Metadata table. This RFC proposes to store Vector Indexes under the Hudi Metadata table similar to Record and Secondary indexes.
+
+So, the vector index related information will be stored under metadata folder.
+
+### Key considerations:
+
+#### Clusters of Indexes (Horizontal Scalability for Vector Indexing)
+
+Approximate Nearest Neighbor (ANN) algorithms are graph-based structures that are inherently designed for in-memory computation. As a result, traditional ANN implementations typically achieve scalability through vertical scaling — by adding more memory and CPU resources to a single node.

Review Comment:
   If a base file contains multiple embedding columns, then yes I would assume that you can have indexes built on each one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-3543815487

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9576",
       "triggerID" : "945a8ffa92f6ebd5a4050d5d2cb2641f079a3161",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 945a8ffa92f6ebd5a4050d5d2cb2641f079a3161 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9576) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] docs: RFC-103 - Support Vector Indexing in Apache Hudi [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #14255:
URL: https://github.com/apache/hudi/pull/14255#issuecomment-5558544590

   RFC-103 has been claimed. There is a similar vector indexing RFC by another author here: https://github.com/apache/hudi/pull/19309
   
   I am going to close this PR as it has been superseded by [RFC-109](#19309). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org