[GitHub] [lucene] markrmiller commented on issue #11507: Increase the number of dims for KNN vectors to 2048 [LUCENE-10471]

["markrmiller (via GitHub)" <gi...@apache.org>]

markrmiller commented on issue #11507:
URL: https://github.com/apache/lucene/issues/11507#issuecomment-1551779674

   While this is not a critique on Lucene's attempt to utilize SIMD via OpenJDK, or any proposed ideas here, it's challenging to envision Lucene emerging as the leading solution for large-scale vector similarity search. This doesn't necessarily imply whether Lucene should or should not integrate such a feature. However, if one were to suggest that this is a critical issue for Lucene's survival, I would question the likelihood of a Java-based engine, laden with keyword search complexities, rising to the top as a vector search solution, regardless of SIMD integration. I would hesitate to wager on its ability to compete with systems exclusively focused on vector similarity search, equipped with first-rate GPU support, and designed to work with existing and future AI-oriented hardware. None of these systems would be developed in Java, nor would they compete with Lucene in the realm of traditional search.
   
   Though it might be beneficial and convenient for Lucene to accommodate this feature, unless the project undergoes a complete overhaul, its survival will likely hinge on the success or failure of its keyword search and faceting capabilities, along with other related features. It appears to be a significant jump to discard all these features into the 'COBOL pile' due to the integration of embeddings. A more plausible scenario is that they will coexist harmoniously, complementing each other's strengths.
   
   Nice little rewrite ChatGPT did there.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org