You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Khaled Alkhouli (Jira)" <ji...@apache.org> on 2025/02/20 13:26:00 UTC

[jira] [Created] (SOLR-17679) Request for Documentation on Hybrid Lexical and Vector Search with Score Breakdown and Cutoff Logic

Khaled Alkhouli created SOLR-17679:
--------------------------------------

             Summary: Request for Documentation on Hybrid Lexical and Vector Search with Score Breakdown and Cutoff Logic
                 Key: SOLR-17679
                 URL: https://issues.apache.org/jira/browse/SOLR-17679
             Project: Solr
          Issue Type: Task
          Components: search
    Affects Versions: 9.6.1
            Reporter: Khaled Alkhouli


Hello Apache Solr team,

I am building a hybrid search engine that combines lexical search (traditional keyword-based search) and vector search (semantic search using embeddings) in a single request. I’m aiming to achieve the following in one request:
 # *Lexical Search:* Using edismax with specified fields and weights.
 # *Vector Search:* Using K-Nearest Neighbors (KNN) based on embeddings.
 # *Hybrid Score Combination:* The final score is the sum of the normalized lexical score and the vector search score. If a document appears in only one search, the other score should be treated as zero.

I have implemented the following logic using Python:
{{}}
{code:java}
def hybrid_search(query, top_k=10):
embedding = np.array(embed([query]), dtype=np.float32)     embedding = list(embedding[0])     lxq = rf"""{{!type=edismax                  qf='all_txt'                 q.op=OR                 tie=0.1             }}({query_terms})"""     solr_query = {         "params": {             "q": "{!bool filter=$retrievalStage must=$rankingStage}",             "rankingStage": "{!func}sum(query($normalisedLexicalQuery),query($vectorQuery))",             "retrievalStage":"{!bool should=$lexicalQuery should=$vectorQuery}", # Union             "normalisedLexicalQuery": "{!func}scale(query($lexicalQuery),0,1)",             "lexicalQuery": lxq,             "vectorQuery": f"{{!knn f=all_v512 topK={top_k}}}{embedding}",             "fl": "post_id,all_txt,score",             "rows": top_k,             "fq": [""],             "rq": "{!rerank reRankQuery=$rqq reRankDocs=100 reRankWeight=3}",             "rqq": "{!frange l=$cutoff}query($rankingStage)",             "sort": "score desc",             "cutoff": f"{cutoff_ratio}"         }     }     response = requests.post(SOLR_URL, headers=HEADERS, json=solr_query)     response = response.json()     return response
{code}
The response returns documents with a combined score, which I assume is the addition of:
 * *Lexical Search Score:* Normalized between 0 and 1.
 * *Vector Search Score:* Already bounded between 0 and 1.

If a document is present in one search but not the other, the score from the missing part is added as zero.
h3. *Request:*

I would like documentation or guidance on the following:
 # *View and Return Individual Scores:*
How can I retrieve the following scores in the same request?

 ** Lexical search score
 ** Vector search score
 ** Final combined score (already retrieved)
I would like to display all three scores in the response together for each document.

 # *Cutoff Logic:*
I am using a Python function to calculate a cutoff threshold based on the scores. Is it possible to implement this cutoff directly in Solr so that only documents that pass a certain threshold are returned? If so, how can I achieve this within Solr’s query syntax, without relying on external Python logic?

I appreciate any help or documentation that can assist with:
 * Returning separate scores for lexical and vector queries.
 * Implementing cutoff logic natively in Solr.

Thank you!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org