[GitHub] [lucene] rmuir commented on pull request #728: LUCENE-10194 Buffer KNN vectors on disk

[GitBox <gi...@apache.org>]

rmuir commented on pull request #728:
URL: https://github.com/apache/lucene/pull/728#issuecomment-1058815901


   Sorry I see it differently.
   
   I'm not a fan of IndexWriter handling the temporary files/encoding/decoding data, this seems to be in the wrong place.
   
   If IndexWriter shouldn't buffer vectors, then can it simply stream vectors to the codec api? This would be similar to how StoredFields and TermVectors work today (see e.g. StoredFieldsConsumer). 
   
   The problem is, today we have two cases of IndexWriter behavior
   1. Stuff that indexwriter buffers in memory and flushes in batches to the codec (terms, postings, docvalues, etc)
   2. Stuff that indexwriter streams directly to the codec (stored fields, term vectors)
   
   For our own sanity, let's avoid adding a third case :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org