You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Shiva Jahangiri <sj...@scu.edu> on 2026/01/13 22:04:20 UTC
[DISCUSS][APE] Adding Vector Index
Hi all,
Initiating discussion to add vector index in AsterixDB to support
approximate nearest neighbor (ANN) queries.
Feature: Adding Vector Index
Details: Currently AsterixDB does not support approximate nearest queries
and similarity search on vector embeddings. This proposal suggests the
first design of a tree-based vector indexing supporting top-k ANN queries
which is fully compatible with LSM structure of AsterixDB's storage. As
part of this proposal we provide support for :
* Adding vector distance functions to support K-Nearest Neighbor (KNN)
queries
* Adding vector index to support ANN queries
* Adding support for INCLUDE fields in vector index to better support
filtered similarity search.
APE:
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
Thanks,
Shiva
--
Shiva Jahangiri
Assistant Professor in Computer Science and Engineering Department
Santa Clara University
[RESULT] APE 31: Vector Index
Posted by Ian Maxon <im...@apache.org>.
By the expedited APE acceptance process, APE 31 is adopted with the
following PMC members expressing their acceptance of the proposal:
+1's
Mike Carey
Taewoo Kim
Murtadha Hubail
Ian Maxon
Discussion thread:
https://lists.apache.org/thread/7mrtgrrybw88hvfy6f7c7jtk4jkn5b6c
Additionally, the APE is now merged into master, at
https://github.com/apache/asterixdb/commit/4364b18776e6331808c62d1163592c62f30464d9
and the 4 preceding commits representing the initial implementation.
Thanks,
- Ian
Re: [DISCUSS][APE] Adding Vector Index
Posted by Murtadha Hubail <hu...@gmail.com>.
+1
-Murtadha
________________________________
From: Taewoo Kim <wa...@gmail.com>
Sent: Friday, January 30, 2026 11:18:52 PM
To: dev@asterixdb.apache.org <de...@asterixdb.apache.org>
Subject: Re: [DISCUSS][APE] Adding Vector Index
Here's my explicit +1
Best,
Taewoo
On Fri, Jan 30, 2026 at 12:13 PM Mike Carey <dt...@gmail.com> wrote:
> Likewise! +1
>
> On Fri, Jan 30, 2026 at 9:58 AM Ian Maxon <im...@apache.org> wrote:
>
> > +1 to this APE. Supporting ANN queries using a vector index,
> > especially a novel one like this, is awesome.
> > I think there are still some loose ends about the distance functions
> > and some of the exact form of the WITH clauses but these are minor
> > details; I don't think they need to block acceptance.
> >
> > On Wed, Jan 14, 2026 at 11:57 AM Taewoo Kim <wa...@gmail.com> wrote:
> > >
> > > Thanks for the clarification!
> > >
> > > Best,
> > > Taewoo
> > >
> > >
> > > On Wed, Jan 14, 2026 at 11:36 AM Shiva Jahangiri <sj...@scu.edu>
> > wrote:
> > >
> > > > Hi Taewoo,
> > > >
> > > > Thanks! So each data partition will have its own vector index as
> > secondary
> > > > index, and so each data partition does its own sampling of data,
> > creates
> > > > its own static structure, etc. using its own data. That means there
> is
> > no
> > > > overlapping or connection between the data or static structure of
> > vector
> > > > indexes of different data partitions.
> > > >
> > > > For top-k, we basically ask each data partition to give us their
> top-k
> > > > results. If we have N data partitions, we will get N*k results which
> > then
> > > > in a global step we get its top-k results out.
> > > >
> > > > Each node can have multiple data partitions which makes it simple in
> > the
> > > > shared-nothing architecture. In the cloud mode, the code is modified
> to
> > > > make sure that the search goes through all data partitions even if
> > multiple
> > > > of them are managed by a single compute node.
> > > >
> > > > Best,
> > > > Shiva
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Shiva,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > Somehow I got confused about TOP-K. I thought each partition could
> > have
> > > > an
> > > > > overlapping portion from the static part. So, will each partition
> be
> > > > > processed on a single node?
> > > > >
> > > > > Regarding the memory part, I'm glad to know that the size is not
> that
> > > > huge.
> > > > > :-)
> > > > >
> > > > > Best,
> > > > > Taewoo
> > > > >
> > > > >
> > > > > On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <
> sjahangiri@scu.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Taewoo,
> > > > > >
> > > > > > Thanks for the great questions.
> > > > > >
> > > > > > With regard to the distributed top-k, each data partition will
> > return
> > > > its
> > > > > > top-k result (if it has at least k records) and then we get the
> > global
> > > > > > top-k based on these local ones (somewhat similar to group by).
> > > > > >
> > > > > > With regard to the memory usage, the static part has to remain in
> > the
> > > > > > memory and our experiments have showed that its size is not that
> > large
> > > > > > compared to the size of the data (for 18GB of data stored in one
> > data
> > > > > > partition,1 Million records each with an embedding with the size
> > of 960
> > > > > > dimensions the static part takes 11MB of memory). The reason that
> > our
> > > > > index
> > > > > > is not memory hungry is that we only have embeddings in the
> static
> > > > part,
> > > > > > the data pages in the dynamic part where the records will be
> > inserted
> > > > > does
> > > > > > not store the embedding of the record, instead it stores its
> > distance
> > > > to
> > > > > > the cluster’s centroid. We will later on explore storing the
> > quantized
> > > > > > vectors for each record (helps reducing execution time by sending
> > > > lesser
> > > > > > records to the primary index for distance calculations) and that
> > might
> > > > > > change the size of the dynamic section. It is important to note
> > that
> > > > each
> > > > > > time a new memory component is created the static structure is
> > copied
> > > > > into
> > > > > > the memory component and the dynamic part will be filled with the
> > > > > incoming
> > > > > > data.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Shiva
> > > > > >
> > > > > > Shiva Jahangiri
> > > > > > Assistant Professor in Computer Science and Engineering
> Department
> > > > > > Santa Clara University
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Shiva,
> > > > > > >
> > > > > > > This proposal looks good.
> > > > > > >
> > > > > > > I have two questions (sorry if I missed)
> > > > > > >
> > > > > > > How are we going to handle distributed execution when dealing
> > with
> > > > > top-K
> > > > > > > ANN?
> > > > > > > How does the memory component look like in terms of
> configurable
> > > > size?
> > > > > My
> > > > > > > naive understanding is that Vector index itself is
> memory-hungry.
> > > > > > >
> > > > > > > Best,
> > > > > > > Taewoo
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <
> > sjahangiri@scu.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Initiating discussion to add vector index in AsterixDB to
> > support
> > > > > > > > approximate nearest neighbor (ANN) queries.
> > > > > > > >
> > > > > > > > Feature: Adding Vector Index
> > > > > > > >
> > > > > > > > Details: Currently AsterixDB does not support approximate
> > nearest
> > > > > > queries
> > > > > > > > and similarity search on vector embeddings. This proposal
> > suggests
> > > > > the
> > > > > > > > first design of a tree-based vector indexing supporting top-k
> > ANN
> > > > > > queries
> > > > > > > > which is fully compatible with LSM structure of AsterixDB's
> > > > storage.
> > > > > As
> > > > > > > > part of this proposal we provide support for :
> > > > > > > >
> > > > > > > > * Adding vector distance functions to support K-Nearest
> > Neighbor
> > > > > (KNN)
> > > > > > > > queries
> > > > > > > > * Adding vector index to support ANN queries
> > > > > > > > * Adding support for INCLUDE fields in vector index to better
> > > > support
> > > > > > > > filtered similarity search.
> > > > > > > >
> > > > > > > > APE:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Shiva
> > > > > > > >
> > > > > > > > --
> > > > > > > > Shiva Jahangiri
> > > > > > > > Assistant Professor in Computer Science and Engineering
> > Department
> > > > > > > > Santa Clara University
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shiva Jahangiri
> > > > Assistant Professor in Computer Science and Engineering Department
> > > > Santa Clara University
> > > >
> >
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Taewoo Kim <wa...@gmail.com>.
Here's my explicit +1
Best,
Taewoo
On Fri, Jan 30, 2026 at 12:13 PM Mike Carey <dt...@gmail.com> wrote:
> Likewise! +1
>
> On Fri, Jan 30, 2026 at 9:58 AM Ian Maxon <im...@apache.org> wrote:
>
> > +1 to this APE. Supporting ANN queries using a vector index,
> > especially a novel one like this, is awesome.
> > I think there are still some loose ends about the distance functions
> > and some of the exact form of the WITH clauses but these are minor
> > details; I don't think they need to block acceptance.
> >
> > On Wed, Jan 14, 2026 at 11:57 AM Taewoo Kim <wa...@gmail.com> wrote:
> > >
> > > Thanks for the clarification!
> > >
> > > Best,
> > > Taewoo
> > >
> > >
> > > On Wed, Jan 14, 2026 at 11:36 AM Shiva Jahangiri <sj...@scu.edu>
> > wrote:
> > >
> > > > Hi Taewoo,
> > > >
> > > > Thanks! So each data partition will have its own vector index as
> > secondary
> > > > index, and so each data partition does its own sampling of data,
> > creates
> > > > its own static structure, etc. using its own data. That means there
> is
> > no
> > > > overlapping or connection between the data or static structure of
> > vector
> > > > indexes of different data partitions.
> > > >
> > > > For top-k, we basically ask each data partition to give us their
> top-k
> > > > results. If we have N data partitions, we will get N*k results which
> > then
> > > > in a global step we get its top-k results out.
> > > >
> > > > Each node can have multiple data partitions which makes it simple in
> > the
> > > > shared-nothing architecture. In the cloud mode, the code is modified
> to
> > > > make sure that the search goes through all data partitions even if
> > multiple
> > > > of them are managed by a single compute node.
> > > >
> > > > Best,
> > > > Shiva
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Shiva,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > Somehow I got confused about TOP-K. I thought each partition could
> > have
> > > > an
> > > > > overlapping portion from the static part. So, will each partition
> be
> > > > > processed on a single node?
> > > > >
> > > > > Regarding the memory part, I'm glad to know that the size is not
> that
> > > > huge.
> > > > > :-)
> > > > >
> > > > > Best,
> > > > > Taewoo
> > > > >
> > > > >
> > > > > On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <
> sjahangiri@scu.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Taewoo,
> > > > > >
> > > > > > Thanks for the great questions.
> > > > > >
> > > > > > With regard to the distributed top-k, each data partition will
> > return
> > > > its
> > > > > > top-k result (if it has at least k records) and then we get the
> > global
> > > > > > top-k based on these local ones (somewhat similar to group by).
> > > > > >
> > > > > > With regard to the memory usage, the static part has to remain in
> > the
> > > > > > memory and our experiments have showed that its size is not that
> > large
> > > > > > compared to the size of the data (for 18GB of data stored in one
> > data
> > > > > > partition,1 Million records each with an embedding with the size
> > of 960
> > > > > > dimensions the static part takes 11MB of memory). The reason that
> > our
> > > > > index
> > > > > > is not memory hungry is that we only have embeddings in the
> static
> > > > part,
> > > > > > the data pages in the dynamic part where the records will be
> > inserted
> > > > > does
> > > > > > not store the embedding of the record, instead it stores its
> > distance
> > > > to
> > > > > > the cluster’s centroid. We will later on explore storing the
> > quantized
> > > > > > vectors for each record (helps reducing execution time by sending
> > > > lesser
> > > > > > records to the primary index for distance calculations) and that
> > might
> > > > > > change the size of the dynamic section. It is important to note
> > that
> > > > each
> > > > > > time a new memory component is created the static structure is
> > copied
> > > > > into
> > > > > > the memory component and the dynamic part will be filled with the
> > > > > incoming
> > > > > > data.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Shiva
> > > > > >
> > > > > > Shiva Jahangiri
> > > > > > Assistant Professor in Computer Science and Engineering
> Department
> > > > > > Santa Clara University
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Shiva,
> > > > > > >
> > > > > > > This proposal looks good.
> > > > > > >
> > > > > > > I have two questions (sorry if I missed)
> > > > > > >
> > > > > > > How are we going to handle distributed execution when dealing
> > with
> > > > > top-K
> > > > > > > ANN?
> > > > > > > How does the memory component look like in terms of
> configurable
> > > > size?
> > > > > My
> > > > > > > naive understanding is that Vector index itself is
> memory-hungry.
> > > > > > >
> > > > > > > Best,
> > > > > > > Taewoo
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <
> > sjahangiri@scu.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Initiating discussion to add vector index in AsterixDB to
> > support
> > > > > > > > approximate nearest neighbor (ANN) queries.
> > > > > > > >
> > > > > > > > Feature: Adding Vector Index
> > > > > > > >
> > > > > > > > Details: Currently AsterixDB does not support approximate
> > nearest
> > > > > > queries
> > > > > > > > and similarity search on vector embeddings. This proposal
> > suggests
> > > > > the
> > > > > > > > first design of a tree-based vector indexing supporting top-k
> > ANN
> > > > > > queries
> > > > > > > > which is fully compatible with LSM structure of AsterixDB's
> > > > storage.
> > > > > As
> > > > > > > > part of this proposal we provide support for :
> > > > > > > >
> > > > > > > > * Adding vector distance functions to support K-Nearest
> > Neighbor
> > > > > (KNN)
> > > > > > > > queries
> > > > > > > > * Adding vector index to support ANN queries
> > > > > > > > * Adding support for INCLUDE fields in vector index to better
> > > > support
> > > > > > > > filtered similarity search.
> > > > > > > >
> > > > > > > > APE:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Shiva
> > > > > > > >
> > > > > > > > --
> > > > > > > > Shiva Jahangiri
> > > > > > > > Assistant Professor in Computer Science and Engineering
> > Department
> > > > > > > > Santa Clara University
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shiva Jahangiri
> > > > Assistant Professor in Computer Science and Engineering Department
> > > > Santa Clara University
> > > >
> >
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Mike Carey <dt...@gmail.com>.
Likewise! +1
On Fri, Jan 30, 2026 at 9:58 AM Ian Maxon <im...@apache.org> wrote:
> +1 to this APE. Supporting ANN queries using a vector index,
> especially a novel one like this, is awesome.
> I think there are still some loose ends about the distance functions
> and some of the exact form of the WITH clauses but these are minor
> details; I don't think they need to block acceptance.
>
> On Wed, Jan 14, 2026 at 11:57 AM Taewoo Kim <wa...@gmail.com> wrote:
> >
> > Thanks for the clarification!
> >
> > Best,
> > Taewoo
> >
> >
> > On Wed, Jan 14, 2026 at 11:36 AM Shiva Jahangiri <sj...@scu.edu>
> wrote:
> >
> > > Hi Taewoo,
> > >
> > > Thanks! So each data partition will have its own vector index as
> secondary
> > > index, and so each data partition does its own sampling of data,
> creates
> > > its own static structure, etc. using its own data. That means there is
> no
> > > overlapping or connection between the data or static structure of
> vector
> > > indexes of different data partitions.
> > >
> > > For top-k, we basically ask each data partition to give us their top-k
> > > results. If we have N data partitions, we will get N*k results which
> then
> > > in a global step we get its top-k results out.
> > >
> > > Each node can have multiple data partitions which makes it simple in
> the
> > > shared-nothing architecture. In the cloud mode, the code is modified to
> > > make sure that the search goes through all data partitions even if
> multiple
> > > of them are managed by a single compute node.
> > >
> > > Best,
> > > Shiva
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com>
> wrote:
> > >
> > > > Hi Shiva,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > Somehow I got confused about TOP-K. I thought each partition could
> have
> > > an
> > > > overlapping portion from the static part. So, will each partition be
> > > > processed on a single node?
> > > >
> > > > Regarding the memory part, I'm glad to know that the size is not that
> > > huge.
> > > > :-)
> > > >
> > > > Best,
> > > > Taewoo
> > > >
> > > >
> > > > On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <sj...@scu.edu>
> > > > wrote:
> > > >
> > > > > Hi Taewoo,
> > > > >
> > > > > Thanks for the great questions.
> > > > >
> > > > > With regard to the distributed top-k, each data partition will
> return
> > > its
> > > > > top-k result (if it has at least k records) and then we get the
> global
> > > > > top-k based on these local ones (somewhat similar to group by).
> > > > >
> > > > > With regard to the memory usage, the static part has to remain in
> the
> > > > > memory and our experiments have showed that its size is not that
> large
> > > > > compared to the size of the data (for 18GB of data stored in one
> data
> > > > > partition,1 Million records each with an embedding with the size
> of 960
> > > > > dimensions the static part takes 11MB of memory). The reason that
> our
> > > > index
> > > > > is not memory hungry is that we only have embeddings in the static
> > > part,
> > > > > the data pages in the dynamic part where the records will be
> inserted
> > > > does
> > > > > not store the embedding of the record, instead it stores its
> distance
> > > to
> > > > > the cluster’s centroid. We will later on explore storing the
> quantized
> > > > > vectors for each record (helps reducing execution time by sending
> > > lesser
> > > > > records to the primary index for distance calculations) and that
> might
> > > > > change the size of the dynamic section. It is important to note
> that
> > > each
> > > > > time a new memory component is created the static structure is
> copied
> > > > into
> > > > > the memory component and the dynamic part will be filled with the
> > > > incoming
> > > > > data.
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Shiva
> > > > >
> > > > > Shiva Jahangiri
> > > > > Assistant Professor in Computer Science and Engineering Department
> > > > > Santa Clara University
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Shiva,
> > > > > >
> > > > > > This proposal looks good.
> > > > > >
> > > > > > I have two questions (sorry if I missed)
> > > > > >
> > > > > > How are we going to handle distributed execution when dealing
> with
> > > > top-K
> > > > > > ANN?
> > > > > > How does the memory component look like in terms of configurable
> > > size?
> > > > My
> > > > > > naive understanding is that Vector index itself is memory-hungry.
> > > > > >
> > > > > > Best,
> > > > > > Taewoo
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <
> sjahangiri@scu.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Initiating discussion to add vector index in AsterixDB to
> support
> > > > > > > approximate nearest neighbor (ANN) queries.
> > > > > > >
> > > > > > > Feature: Adding Vector Index
> > > > > > >
> > > > > > > Details: Currently AsterixDB does not support approximate
> nearest
> > > > > queries
> > > > > > > and similarity search on vector embeddings. This proposal
> suggests
> > > > the
> > > > > > > first design of a tree-based vector indexing supporting top-k
> ANN
> > > > > queries
> > > > > > > which is fully compatible with LSM structure of AsterixDB's
> > > storage.
> > > > As
> > > > > > > part of this proposal we provide support for :
> > > > > > >
> > > > > > > * Adding vector distance functions to support K-Nearest
> Neighbor
> > > > (KNN)
> > > > > > > queries
> > > > > > > * Adding vector index to support ANN queries
> > > > > > > * Adding support for INCLUDE fields in vector index to better
> > > support
> > > > > > > filtered similarity search.
> > > > > > >
> > > > > > > APE:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Shiva
> > > > > > >
> > > > > > > --
> > > > > > > Shiva Jahangiri
> > > > > > > Assistant Professor in Computer Science and Engineering
> Department
> > > > > > > Santa Clara University
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Shiva Jahangiri
> > > Assistant Professor in Computer Science and Engineering Department
> > > Santa Clara University
> > >
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Ian Maxon <im...@apache.org>.
+1 to this APE. Supporting ANN queries using a vector index,
especially a novel one like this, is awesome.
I think there are still some loose ends about the distance functions
and some of the exact form of the WITH clauses but these are minor
details; I don't think they need to block acceptance.
On Wed, Jan 14, 2026 at 11:57 AM Taewoo Kim <wa...@gmail.com> wrote:
>
> Thanks for the clarification!
>
> Best,
> Taewoo
>
>
> On Wed, Jan 14, 2026 at 11:36 AM Shiva Jahangiri <sj...@scu.edu> wrote:
>
> > Hi Taewoo,
> >
> > Thanks! So each data partition will have its own vector index as secondary
> > index, and so each data partition does its own sampling of data, creates
> > its own static structure, etc. using its own data. That means there is no
> > overlapping or connection between the data or static structure of vector
> > indexes of different data partitions.
> >
> > For top-k, we basically ask each data partition to give us their top-k
> > results. If we have N data partitions, we will get N*k results which then
> > in a global step we get its top-k results out.
> >
> > Each node can have multiple data partitions which makes it simple in the
> > shared-nothing architecture. In the cloud mode, the code is modified to
> > make sure that the search goes through all data partitions even if multiple
> > of them are managed by a single compute node.
> >
> > Best,
> > Shiva
> >
> >
> >
> >
> >
> >
> > On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com> wrote:
> >
> > > Hi Shiva,
> > >
> > > Thanks for your reply.
> > >
> > > Somehow I got confused about TOP-K. I thought each partition could have
> > an
> > > overlapping portion from the static part. So, will each partition be
> > > processed on a single node?
> > >
> > > Regarding the memory part, I'm glad to know that the size is not that
> > huge.
> > > :-)
> > >
> > > Best,
> > > Taewoo
> > >
> > >
> > > On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <sj...@scu.edu>
> > > wrote:
> > >
> > > > Hi Taewoo,
> > > >
> > > > Thanks for the great questions.
> > > >
> > > > With regard to the distributed top-k, each data partition will return
> > its
> > > > top-k result (if it has at least k records) and then we get the global
> > > > top-k based on these local ones (somewhat similar to group by).
> > > >
> > > > With regard to the memory usage, the static part has to remain in the
> > > > memory and our experiments have showed that its size is not that large
> > > > compared to the size of the data (for 18GB of data stored in one data
> > > > partition,1 Million records each with an embedding with the size of 960
> > > > dimensions the static part takes 11MB of memory). The reason that our
> > > index
> > > > is not memory hungry is that we only have embeddings in the static
> > part,
> > > > the data pages in the dynamic part where the records will be inserted
> > > does
> > > > not store the embedding of the record, instead it stores its distance
> > to
> > > > the cluster’s centroid. We will later on explore storing the quantized
> > > > vectors for each record (helps reducing execution time by sending
> > lesser
> > > > records to the primary index for distance calculations) and that might
> > > > change the size of the dynamic section. It is important to note that
> > each
> > > > time a new memory component is created the static structure is copied
> > > into
> > > > the memory component and the dynamic part will be filled with the
> > > incoming
> > > > data.
> > > >
> > > >
> > > >
> > > > Best,
> > > > Shiva
> > > >
> > > > Shiva Jahangiri
> > > > Assistant Professor in Computer Science and Engineering Department
> > > > Santa Clara University
> > > >
> > > >
> > > >
> > > > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com> wrote:
> > > >
> > > > > Hi Shiva,
> > > > >
> > > > > This proposal looks good.
> > > > >
> > > > > I have two questions (sorry if I missed)
> > > > >
> > > > > How are we going to handle distributed execution when dealing with
> > > top-K
> > > > > ANN?
> > > > > How does the memory component look like in terms of configurable
> > size?
> > > My
> > > > > naive understanding is that Vector index itself is memory-hungry.
> > > > >
> > > > > Best,
> > > > > Taewoo
> > > > >
> > > > >
> > > > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Initiating discussion to add vector index in AsterixDB to support
> > > > > > approximate nearest neighbor (ANN) queries.
> > > > > >
> > > > > > Feature: Adding Vector Index
> > > > > >
> > > > > > Details: Currently AsterixDB does not support approximate nearest
> > > > queries
> > > > > > and similarity search on vector embeddings. This proposal suggests
> > > the
> > > > > > first design of a tree-based vector indexing supporting top-k ANN
> > > > queries
> > > > > > which is fully compatible with LSM structure of AsterixDB's
> > storage.
> > > As
> > > > > > part of this proposal we provide support for :
> > > > > >
> > > > > > * Adding vector distance functions to support K-Nearest Neighbor
> > > (KNN)
> > > > > > queries
> > > > > > * Adding vector index to support ANN queries
> > > > > > * Adding support for INCLUDE fields in vector index to better
> > support
> > > > > > filtered similarity search.
> > > > > >
> > > > > > APE:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > > > >
> > > > > > Thanks,
> > > > > > Shiva
> > > > > >
> > > > > > --
> > > > > > Shiva Jahangiri
> > > > > > Assistant Professor in Computer Science and Engineering Department
> > > > > > Santa Clara University
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Shiva Jahangiri
> > Assistant Professor in Computer Science and Engineering Department
> > Santa Clara University
> >
Re: [DISCUSS][APE] Adding Vector Index
Posted by Taewoo Kim <wa...@gmail.com>.
Thanks for the clarification!
Best,
Taewoo
On Wed, Jan 14, 2026 at 11:36 AM Shiva Jahangiri <sj...@scu.edu> wrote:
> Hi Taewoo,
>
> Thanks! So each data partition will have its own vector index as secondary
> index, and so each data partition does its own sampling of data, creates
> its own static structure, etc. using its own data. That means there is no
> overlapping or connection between the data or static structure of vector
> indexes of different data partitions.
>
> For top-k, we basically ask each data partition to give us their top-k
> results. If we have N data partitions, we will get N*k results which then
> in a global step we get its top-k results out.
>
> Each node can have multiple data partitions which makes it simple in the
> shared-nothing architecture. In the cloud mode, the code is modified to
> make sure that the search goes through all data partitions even if multiple
> of them are managed by a single compute node.
>
> Best,
> Shiva
>
>
>
>
>
>
> On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com> wrote:
>
> > Hi Shiva,
> >
> > Thanks for your reply.
> >
> > Somehow I got confused about TOP-K. I thought each partition could have
> an
> > overlapping portion from the static part. So, will each partition be
> > processed on a single node?
> >
> > Regarding the memory part, I'm glad to know that the size is not that
> huge.
> > :-)
> >
> > Best,
> > Taewoo
> >
> >
> > On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <sj...@scu.edu>
> > wrote:
> >
> > > Hi Taewoo,
> > >
> > > Thanks for the great questions.
> > >
> > > With regard to the distributed top-k, each data partition will return
> its
> > > top-k result (if it has at least k records) and then we get the global
> > > top-k based on these local ones (somewhat similar to group by).
> > >
> > > With regard to the memory usage, the static part has to remain in the
> > > memory and our experiments have showed that its size is not that large
> > > compared to the size of the data (for 18GB of data stored in one data
> > > partition,1 Million records each with an embedding with the size of 960
> > > dimensions the static part takes 11MB of memory). The reason that our
> > index
> > > is not memory hungry is that we only have embeddings in the static
> part,
> > > the data pages in the dynamic part where the records will be inserted
> > does
> > > not store the embedding of the record, instead it stores its distance
> to
> > > the cluster’s centroid. We will later on explore storing the quantized
> > > vectors for each record (helps reducing execution time by sending
> lesser
> > > records to the primary index for distance calculations) and that might
> > > change the size of the dynamic section. It is important to note that
> each
> > > time a new memory component is created the static structure is copied
> > into
> > > the memory component and the dynamic part will be filled with the
> > incoming
> > > data.
> > >
> > >
> > >
> > > Best,
> > > Shiva
> > >
> > > Shiva Jahangiri
> > > Assistant Professor in Computer Science and Engineering Department
> > > Santa Clara University
> > >
> > >
> > >
> > > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com> wrote:
> > >
> > > > Hi Shiva,
> > > >
> > > > This proposal looks good.
> > > >
> > > > I have two questions (sorry if I missed)
> > > >
> > > > How are we going to handle distributed execution when dealing with
> > top-K
> > > > ANN?
> > > > How does the memory component look like in terms of configurable
> size?
> > My
> > > > naive understanding is that Vector index itself is memory-hungry.
> > > >
> > > > Best,
> > > > Taewoo
> > > >
> > > >
> > > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Initiating discussion to add vector index in AsterixDB to support
> > > > > approximate nearest neighbor (ANN) queries.
> > > > >
> > > > > Feature: Adding Vector Index
> > > > >
> > > > > Details: Currently AsterixDB does not support approximate nearest
> > > queries
> > > > > and similarity search on vector embeddings. This proposal suggests
> > the
> > > > > first design of a tree-based vector indexing supporting top-k ANN
> > > queries
> > > > > which is fully compatible with LSM structure of AsterixDB's
> storage.
> > As
> > > > > part of this proposal we provide support for :
> > > > >
> > > > > * Adding vector distance functions to support K-Nearest Neighbor
> > (KNN)
> > > > > queries
> > > > > * Adding vector index to support ANN queries
> > > > > * Adding support for INCLUDE fields in vector index to better
> support
> > > > > filtered similarity search.
> > > > >
> > > > > APE:
> > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > > >
> > > > > Thanks,
> > > > > Shiva
> > > > >
> > > > > --
> > > > > Shiva Jahangiri
> > > > > Assistant Professor in Computer Science and Engineering Department
> > > > > Santa Clara University
> > > > >
> > > >
> > >
> >
>
>
> --
> Shiva Jahangiri
> Assistant Professor in Computer Science and Engineering Department
> Santa Clara University
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Shiva Jahangiri <sj...@scu.edu>.
Hi Taewoo,
Thanks! So each data partition will have its own vector index as secondary
index, and so each data partition does its own sampling of data, creates
its own static structure, etc. using its own data. That means there is no
overlapping or connection between the data or static structure of vector
indexes of different data partitions.
For top-k, we basically ask each data partition to give us their top-k
results. If we have N data partitions, we will get N*k results which then
in a global step we get its top-k results out.
Each node can have multiple data partitions which makes it simple in the
shared-nothing architecture. In the cloud mode, the code is modified to
make sure that the search goes through all data partitions even if multiple
of them are managed by a single compute node.
Best,
Shiva
On Wed, Jan 14, 2026 at 10:39 AM Taewoo Kim <wa...@gmail.com> wrote:
> Hi Shiva,
>
> Thanks for your reply.
>
> Somehow I got confused about TOP-K. I thought each partition could have an
> overlapping portion from the static part. So, will each partition be
> processed on a single node?
>
> Regarding the memory part, I'm glad to know that the size is not that huge.
> :-)
>
> Best,
> Taewoo
>
>
> On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <sj...@scu.edu>
> wrote:
>
> > Hi Taewoo,
> >
> > Thanks for the great questions.
> >
> > With regard to the distributed top-k, each data partition will return its
> > top-k result (if it has at least k records) and then we get the global
> > top-k based on these local ones (somewhat similar to group by).
> >
> > With regard to the memory usage, the static part has to remain in the
> > memory and our experiments have showed that its size is not that large
> > compared to the size of the data (for 18GB of data stored in one data
> > partition,1 Million records each with an embedding with the size of 960
> > dimensions the static part takes 11MB of memory). The reason that our
> index
> > is not memory hungry is that we only have embeddings in the static part,
> > the data pages in the dynamic part where the records will be inserted
> does
> > not store the embedding of the record, instead it stores its distance to
> > the cluster’s centroid. We will later on explore storing the quantized
> > vectors for each record (helps reducing execution time by sending lesser
> > records to the primary index for distance calculations) and that might
> > change the size of the dynamic section. It is important to note that each
> > time a new memory component is created the static structure is copied
> into
> > the memory component and the dynamic part will be filled with the
> incoming
> > data.
> >
> >
> >
> > Best,
> > Shiva
> >
> > Shiva Jahangiri
> > Assistant Professor in Computer Science and Engineering Department
> > Santa Clara University
> >
> >
> >
> > On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com> wrote:
> >
> > > Hi Shiva,
> > >
> > > This proposal looks good.
> > >
> > > I have two questions (sorry if I missed)
> > >
> > > How are we going to handle distributed execution when dealing with
> top-K
> > > ANN?
> > > How does the memory component look like in terms of configurable size?
> My
> > > naive understanding is that Vector index itself is memory-hungry.
> > >
> > > Best,
> > > Taewoo
> > >
> > >
> > > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Initiating discussion to add vector index in AsterixDB to support
> > > > approximate nearest neighbor (ANN) queries.
> > > >
> > > > Feature: Adding Vector Index
> > > >
> > > > Details: Currently AsterixDB does not support approximate nearest
> > queries
> > > > and similarity search on vector embeddings. This proposal suggests
> the
> > > > first design of a tree-based vector indexing supporting top-k ANN
> > queries
> > > > which is fully compatible with LSM structure of AsterixDB's storage.
> As
> > > > part of this proposal we provide support for :
> > > >
> > > > * Adding vector distance functions to support K-Nearest Neighbor
> (KNN)
> > > > queries
> > > > * Adding vector index to support ANN queries
> > > > * Adding support for INCLUDE fields in vector index to better support
> > > > filtered similarity search.
> > > >
> > > > APE:
> > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > > >
> > > > Thanks,
> > > > Shiva
> > > >
> > > > --
> > > > Shiva Jahangiri
> > > > Assistant Professor in Computer Science and Engineering Department
> > > > Santa Clara University
> > > >
> > >
> >
>
--
Shiva Jahangiri
Assistant Professor in Computer Science and Engineering Department
Santa Clara University
Re: [DISCUSS][APE] Adding Vector Index
Posted by Taewoo Kim <wa...@gmail.com>.
Hi Shiva,
Thanks for your reply.
Somehow I got confused about TOP-K. I thought each partition could have an
overlapping portion from the static part. So, will each partition be
processed on a single node?
Regarding the memory part, I'm glad to know that the size is not that huge.
:-)
Best,
Taewoo
On Wed, Jan 14, 2026 at 9:25 AM Shiva Jahangiri <sj...@scu.edu> wrote:
> Hi Taewoo,
>
> Thanks for the great questions.
>
> With regard to the distributed top-k, each data partition will return its
> top-k result (if it has at least k records) and then we get the global
> top-k based on these local ones (somewhat similar to group by).
>
> With regard to the memory usage, the static part has to remain in the
> memory and our experiments have showed that its size is not that large
> compared to the size of the data (for 18GB of data stored in one data
> partition,1 Million records each with an embedding with the size of 960
> dimensions the static part takes 11MB of memory). The reason that our index
> is not memory hungry is that we only have embeddings in the static part,
> the data pages in the dynamic part where the records will be inserted does
> not store the embedding of the record, instead it stores its distance to
> the cluster’s centroid. We will later on explore storing the quantized
> vectors for each record (helps reducing execution time by sending lesser
> records to the primary index for distance calculations) and that might
> change the size of the dynamic section. It is important to note that each
> time a new memory component is created the static structure is copied into
> the memory component and the dynamic part will be filled with the incoming
> data.
>
>
>
> Best,
> Shiva
>
> Shiva Jahangiri
> Assistant Professor in Computer Science and Engineering Department
> Santa Clara University
>
>
>
> On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com> wrote:
>
> > Hi Shiva,
> >
> > This proposal looks good.
> >
> > I have two questions (sorry if I missed)
> >
> > How are we going to handle distributed execution when dealing with top-K
> > ANN?
> > How does the memory component look like in terms of configurable size? My
> > naive understanding is that Vector index itself is memory-hungry.
> >
> > Best,
> > Taewoo
> >
> >
> > On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu>
> > wrote:
> >
> > > Hi all,
> > >
> > > Initiating discussion to add vector index in AsterixDB to support
> > > approximate nearest neighbor (ANN) queries.
> > >
> > > Feature: Adding Vector Index
> > >
> > > Details: Currently AsterixDB does not support approximate nearest
> queries
> > > and similarity search on vector embeddings. This proposal suggests the
> > > first design of a tree-based vector indexing supporting top-k ANN
> queries
> > > which is fully compatible with LSM structure of AsterixDB's storage. As
> > > part of this proposal we provide support for :
> > >
> > > * Adding vector distance functions to support K-Nearest Neighbor (KNN)
> > > queries
> > > * Adding vector index to support ANN queries
> > > * Adding support for INCLUDE fields in vector index to better support
> > > filtered similarity search.
> > >
> > > APE:
> > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> > >
> > > Thanks,
> > > Shiva
> > >
> > > --
> > > Shiva Jahangiri
> > > Assistant Professor in Computer Science and Engineering Department
> > > Santa Clara University
> > >
> >
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Shiva Jahangiri <sj...@scu.edu>.
Hi Taewoo,
Thanks for the great questions.
With regard to the distributed top-k, each data partition will return its
top-k result (if it has at least k records) and then we get the global
top-k based on these local ones (somewhat similar to group by).
With regard to the memory usage, the static part has to remain in the
memory and our experiments have showed that its size is not that large
compared to the size of the data (for 18GB of data stored in one data
partition,1 Million records each with an embedding with the size of 960
dimensions the static part takes 11MB of memory). The reason that our index
is not memory hungry is that we only have embeddings in the static part,
the data pages in the dynamic part where the records will be inserted does
not store the embedding of the record, instead it stores its distance to
the cluster’s centroid. We will later on explore storing the quantized
vectors for each record (helps reducing execution time by sending lesser
records to the primary index for distance calculations) and that might
change the size of the dynamic section. It is important to note that each
time a new memory component is created the static structure is copied into
the memory component and the dynamic part will be filled with the incoming
data.
Best,
Shiva
Shiva Jahangiri
Assistant Professor in Computer Science and Engineering Department
Santa Clara University
On Tue, Jan 13, 2026 at 3:24 PM Taewoo Kim <wa...@gmail.com> wrote:
> Hi Shiva,
>
> This proposal looks good.
>
> I have two questions (sorry if I missed)
>
> How are we going to handle distributed execution when dealing with top-K
> ANN?
> How does the memory component look like in terms of configurable size? My
> naive understanding is that Vector index itself is memory-hungry.
>
> Best,
> Taewoo
>
>
> On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu>
> wrote:
>
> > Hi all,
> >
> > Initiating discussion to add vector index in AsterixDB to support
> > approximate nearest neighbor (ANN) queries.
> >
> > Feature: Adding Vector Index
> >
> > Details: Currently AsterixDB does not support approximate nearest queries
> > and similarity search on vector embeddings. This proposal suggests the
> > first design of a tree-based vector indexing supporting top-k ANN queries
> > which is fully compatible with LSM structure of AsterixDB's storage. As
> > part of this proposal we provide support for :
> >
> > * Adding vector distance functions to support K-Nearest Neighbor (KNN)
> > queries
> > * Adding vector index to support ANN queries
> > * Adding support for INCLUDE fields in vector index to better support
> > filtered similarity search.
> >
> > APE:
> >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
> >
> > Thanks,
> > Shiva
> >
> > --
> > Shiva Jahangiri
> > Assistant Professor in Computer Science and Engineering Department
> > Santa Clara University
> >
>
Re: [DISCUSS][APE] Adding Vector Index
Posted by Taewoo Kim <wa...@gmail.com>.
Hi Shiva,
This proposal looks good.
I have two questions (sorry if I missed)
How are we going to handle distributed execution when dealing with top-K
ANN?
How does the memory component look like in terms of configurable size? My
naive understanding is that Vector index itself is memory-hungry.
Best,
Taewoo
On Tue, Jan 13, 2026 at 2:04 PM Shiva Jahangiri <sj...@scu.edu> wrote:
> Hi all,
>
> Initiating discussion to add vector index in AsterixDB to support
> approximate nearest neighbor (ANN) queries.
>
> Feature: Adding Vector Index
>
> Details: Currently AsterixDB does not support approximate nearest queries
> and similarity search on vector embeddings. This proposal suggests the
> first design of a tree-based vector indexing supporting top-k ANN queries
> which is fully compatible with LSM structure of AsterixDB's storage. As
> part of this proposal we provide support for :
>
> * Adding vector distance functions to support K-Nearest Neighbor (KNN)
> queries
> * Adding vector index to support ANN queries
> * Adding support for INCLUDE fields in vector index to better support
> filtered similarity search.
>
> APE:
>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*31*3A*Vector*Index__;KyUrKw!!MLMg-p0Z!D1Zmu-mN_byA8sV_P_p7aLlbYJIg0b19njsPeaMkVTovtWeW0IsD-CIgOo0MJ7_7t3pZsk63GqP6lfK1$
>
> Thanks,
> Shiva
>
> --
> Shiva Jahangiri
> Assistant Professor in Computer Science and Engineering Department
> Santa Clara University
>