[PR] [exploratory] modules/language-models: semantic highlighting component [solr]

["cpoerschke (via GitHub)" <gi...@apache.org>]

cpoerschke opened a new pull request, #4230:
URL: https://github.com/apache/solr/pull/4230

   done-ish list:
    * [X] skeleton `SemanticHighlightingComponent` class in `modules/language-models` to avoid `core/HighlightingComponent.java` having `modules/language-models` dependency
    * [X] created `CustomModel.java` and `custom-model.json` providing hard-coded mock embeddings for test use without an external model provider dependency
    * [X] skeleton logic to use a language model to compute a score for a `Passage`
    * [X] minimal `SemanticHighlightingComponentTest` class to illustrate usage
   
   to-do list, non-exhaustive:
    * [ ] consideration of parameter details e.g. how to request semantic highlighting and the model(s) to use
    * [ ] how to obtain the vector against which passages are compared e.g. from some new parameter directly or by extraction from the `q` or `hl.q` parameter if a knn parser is used or by some other way?
    * [ ] consideration of `Passage` extraction e.g. currently it is term based but what if the `q` was a vector query i.e. no terms
    * [ ] should the `PassageScorer` and/or the `Comparator<Passage>` apply the language model to candidate passages?
    * [ ] how to properly and efficiently compute vector distances? currently using euclidian distance for illustration only.
    * [ ] tests
    * [ ] documentation
    * [ ] ???


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org