You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "yatriks (via GitHub)" <gi...@apache.org> on 2026/03/16 00:34:13 UTC
[D] Question about FT.SEARCH indices [kvrocks]
GitHub user yatriks created a discussion: Question about FT.SEARCH indices
Hello,
I know this is a bit strange of a question, so please bear with me.
Right now, to experiment with the KQIR, I'm using an index on 3 fields, `d TAG, s NUMERIC, e NUMERIC`. All my queries are range queries on `s` and `e` for a particular `d` (There are thousands of keys with `d`, all sharing a prefix). This is really fast generally, but as I'm at around ~100,000 `d`, the search command is starting to get slower.
This database will likely need to be shared anyway is it grows from 100k -> 1M unique values of `d`, and I think a relatively simple `ZSET` structure would work well in that case anyway.
_But for the sake of learning about the internal architecture_, I'm curious -- if I create an index per `d` with a `PREFIX`, I don't think there are any hard limitations, but architecturally what are the issues with this? What, if any, is the issue with 100k indexes that are distinct on PREFIX? 1M? Where are the technical bottlenecks?
Thanks a lot for your time and help :pray:
GitHub link: https://github.com/apache/kvrocks/discussions/3391
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
GitHub user PragmaTwice added a comment to the discussion: Question about FT.SEARCH indices
```
project *: (filter (and e >= T, s < T): tag-scan d, VAL)
```
means it will use the index of `d`.
it wil jump to the first key that `d == VAL`, and scan over it until the last key that `d == VAL`, and filter out via the condition `(and e >= T, s < T)`.
GitHub link: https://github.com/apache/kvrocks/discussions/3391#discussioncomment-16194516
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "yatriks (via GitHub)" <gi...@apache.org>.
GitHub user yatriks reopened a discussion: Question about FT.SEARCH indices
Hello,
I know this is a bit strange of a question, so please bear with me.
Right now, to experiment with the KQIR, I'm using an index on 3 fields, `d TAG, s NUMERIC, e NUMERIC`. All my queries are range queries on `s` and `e` for a particular `d` (There are thousands of keys with `d`, all sharing a prefix). This is really fast generally, but as I'm at around ~100,000 `d`, the search command is starting to get slower.
This database will likely need to be shared anyway is it grows from 100k -> 1M unique values of `d`, and I think a relatively simple `ZSET` structure would work well in that case anyway.
_But for the sake of learning about the internal architecture_, I'm curious -- if I create an index per `d` with a `PREFIX`, I don't think there are any hard limitations, but architecturally what are the issues with this? What, if any, is the issue with 100k indexes that are distinct on PREFIX? 1M? Where are the technical bottlenecks?
Thanks a lot for your time and help :pray:
GitHub link: https://github.com/apache/kvrocks/discussions/3391
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "yatriks (via GitHub)" <gi...@apache.org>.
GitHub user yatriks added a comment to the discussion: Question about FT.SEARCH indices
This makes sense, thank you :pray: and thanks for walking me through the query plan.
GitHub link: https://github.com/apache/kvrocks/discussions/3391#discussioncomment-16197748
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "yatriks (via GitHub)" <gi...@apache.org>.
GitHub user yatriks edited a comment on the discussion: Question about FT.SEARCH indices
Thank you for the response 🙏
Tha queries are always of the form `select d where d==VAL and s<=t and e>t` (sometimes query for multiple t chained with `or`).
Explain is
```
project *: (filter (and e >= T, s < T): tag-scan d, VAL)
```
s, e, and t are floats. Is that doing the tag scan first?
The fixed `d` in the query, as well as the fact that I can use PREFIX to select all keys for `d` is why I was curious about multiple indexes
Thanks again for your time
GitHub link: https://github.com/apache/kvrocks/discussions/3391#discussioncomment-16188519
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "yatriks (via GitHub)" <gi...@apache.org>.
GitHub user yatriks added a comment to the discussion: Question about FT.SEARCH indices
Thank you for the response 🙏
Tha queries are always of the form `select d where d==VAL and s<=t and e>t` (sometimes query for multiple t chained with `or`).
Explain is
```
project *: (filter (and e >= T, s < T): tag-scan d, VAL)
```
s, e, and t are floats. Is that doing the tag scan first?
The fixed `d` in the query, as well as the fact that I can use PREFIX to select all keys for `d` is why I was curious about multiple indexes
GitHub link: https://github.com/apache/kvrocks/discussions/3391#discussioncomment-16188519
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
GitHub user PragmaTwice closed the discussion with a comment: Question about FT.SEARCH indices
Sorry for the late reply.
Could you provide some real data and queries? From the description alone, it's hard to fully understand what queries you're running and what your data looks like.
As things stand, for the three fields `d TAG, s NUMERIC, e NUMERIC` -- if you haven't declared `NOINDEX`, Kvrocks Search will create three separate indexes, one for each field. The index selection algorithm then tries to pick the most suitable one.
There are two shortcomings in the current implementation:
- First, the cost model for index selection is static -- it doesn't account for the actual distribution of your data, which can lead to suboptimal index choices. I've considered letting users provide hints for better selection, but haven't had time to implement that yet.
- Second, composite indexes are not yet supported. If your query is something like `WHERE d == .. AND s == ..`, a composite index would make it extremely fast, instead of the current approach of filtering through one index and then scanning.
You can use `FT.EXPLAINSQL` to see what the current query plan looks like.
GitHub link: https://github.com/apache/kvrocks/discussions/3391#discussioncomment-16186587
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org
Re: [D] Question about FT.SEARCH indices [kvrocks]
Posted by "yatriks (via GitHub)" <gi...@apache.org>.
GitHub user yatriks closed a discussion: Question about FT.SEARCH indices
Hello,
I know this is a bit strange of a question, so please bear with me.
Right now, to experiment with the KQIR, I'm using an index on 3 fields, `d TAG, s NUMERIC, e NUMERIC`. All my queries are range queries on `s` and `e` for a particular `d` (There are thousands of keys with `d`, all sharing a prefix). This is really fast generally, but as I'm at around ~100,000 `d`, the search command is starting to get slower.
This database will likely need to be shared anyway is it grows from 100k -> 1M unique values of `d`, and I think a relatively simple `ZSET` structure would work well in that case anyway.
_But for the sake of learning about the internal architecture_, I'm curious -- if I create an index per `d` with a `PREFIX`, I don't think there are any hard limitations, but architecturally what are the issues with this? What, if any, is the issue with 100k indexes that are distinct on PREFIX? 1M? Where are the technical bottlenecks?
Thanks a lot for your time and help :pray:
GitHub link: https://github.com/apache/kvrocks/discussions/3391
----
This is an automatically sent email for issues@kvrocks.apache.org.
To unsubscribe, please send an email to: issues-unsubscribe@kvrocks.apache.org