[jira] [Commented] (IMPALA-12467) Semantic Search In Impala

["Michael Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881084#comment-17881084 ] 

Michael Smith commented on IMPALA-12467:
----------------------------------------

There's a lot of potential breadth here:
* cosine distance is based on a vector, and implies the selection of an embedding model to turn text into a vector. These tend to include some semantic relationship between words, and would be necessary to satisfy something like "Europe" returning countries or regions in Europe. Handling the embedding calculation as part of the function API would be a lot more useful, but works best with access to GPUs.
* Jaro Winkler, Levenshtein, Hamming all operate directly on strings and don't require a separate representation. They also don't encode meaning, so primarily useful for handling typos or different forms of the same word (quick vs quickly).

I would address those are at least 3 separate problems:
* string similarity functions (like Levenshtein)
* vector distance functions (which can be used on vector embeddings)
* functions to generate vector embeddings from a string using a configured embedding model (and necessary infrastructure)

> Semantic Search In Impala
> -------------------------
>
>                 Key: IMPALA-12467
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12467
>             Project: IMPALA
>          Issue Type: Wish
>            Reporter: Sreenath
>            Priority: Major
>
> _Semantic search is the tech power *vector databases,* and we can have the same power in Impala._
> Semantic search is a way for computers to understand the meaning behind words and phrases when you're searching for something. Instead of just looking for exact matches of keywords, it tries to figure out what you're really asking and provides results that are more relevant and meaningful to your question. It's like having a search engine that can understand what you mean, not just what you say, making it easier to find the information you're looking for. This ticket is a wish to have semantic search in Impala.
> On the implementation side, semantic search uses an embedding model and any of the similarity distance functions.
> My proposal is to implement functions for on-the-fly calculation of similarity distance between two values. Once we have them we could easily do semantic search as part of a where clause.
>  * Eg (using a cosine similarity function): “WHERE cos_dist(region, 'europe') > 0.9“. And it could return records with regions like Scandinavia, Nordic, Baltic etc…
>  * We could have functions thats accept values as text or as vector embeddings.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org