You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "chenboat (via GitHub)" <gi...@apache.org> on 2025/07/07 22:18:23 UTC
[I] [PEP Proposal] Adding ngram index to speed up text search queries with LIKE [pinot]
chenboat opened a new issue, #16294:
URL: https://github.com/apache/pinot/issues/16294
n-gram (https://en.wikipedia.org/wiki/N-gram) index can be applied to string columns to speed up queries with LIKE filtering condition (e.g., LIKE('%pino%')). The basic idea is to extract consecutive character sequences (e.g., _pin_, _ino_, _not_ for n =3 on the string _pinot_) from origin strings and build inverted index on these sequences. When processing LIKE filters for a sub-string match, one can break down the substring into multiple grams similar to the indexing process and then look for matching documents which contains ALL the grams. The remaining documents (often much fewer) are then string matched to validate the exact matching. In some cases where the search string is shorter or equal to the ngram, the final validation can be omitted. Ngram is similar to bloom filtering: both provide effective pruning of non-matching documents.
ngram index is available to open source libraries like [ElasticSearch](https://www.elastic.co/docs/reference/text-analysis/analysis-ngram-tokenizer) and [StarRocks](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index/).
Compared with text index, n-gram index is bigger in size in general because it extracts sub-word sequences. On the other hand, it can process wildcard queries (e.g., %pino%) more efficiently -- note that libraries like Lucene usually discourage leading * in their text queries ([ref](https://lucene.apache.org/core/2_9_4/queryparsersyntax.html#Wildcard%20Searches).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [PEP Proposal] Adding ngram index to speed up text search queries with LIKE [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16294:
URL: https://github.com/apache/pinot/issues/16294#issuecomment-4323435781
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [PEP Proposal] Adding ngram index to speed up text search queries with LIKE [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16294:
URL: https://github.com/apache/pinot/issues/16294#issuecomment-4113147257
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [PEP Proposal] Adding ngram index to speed up text search queries with LIKE [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16294: [PEP Proposal] Adding ngram index to speed up text search queries with LIKE
URL: https://github.com/apache/pinot/issues/16294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org