[I] Are we undercounting RAM used by vectors buffered in an in-memory segment? [lucene]

["mikemccand (via GitHub)" <gi...@apache.org>]

mikemccand opened a new issue, #15901:
URL: https://github.com/apache/lucene/issues/15901

   ### Description
   
   I'm looking at a fun segment trace of a nearly pure flat (no HNSW graph) vectors Lucene index (production index for Amazon product search) and see weirdly that the flushed segments show ~112% RAM efficiency.
   
   RAM efficiency measures size-on-disk of the newly flushed segment against size-in-ram before it was flushed.  For postings heavy indices it's usually poor, maybe 20-30%, because in-RAM postings storage has overhead because they need to grow/realloc as new documents are indexed, appending integers to tons of tiny `int[]` virtual slices (one or two or three per unique term in the document being inverted).  On disk those become well packed.
   
   But for flat vectors, I think we just buffer the vectors in RAM until flush, so the RAM accounting ought to be simple?
   
   This is with 7-bit quantization (aside: why do we now have signed 7 bit, and unsigned 8 bit, `KnnVectorsFormat` options?  is there really a difference?  oh hmm the javadocs says its for backwards compatibility ... shouldn't it (7 bit) be marked deprecated (new Lucene indices shouldn't choose it)?), and @msokolov pointed out we increase index size to store the one-byte-per-dimension quantized vectors in addition to the `float32` vectors.  But then I would have expected 125% RAM efficiency?  So I'm confused why I see halfway (~112%)...
   
   <img width="730" height="530" alt="Image" src="https://github.com/user-attachments/assets/589ac3c8-b390-415f-8c8d-2f6ec68d752c" />
   
   ### Version and environment details
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org

Re: [I] Are we undercounting RAM used by vectors buffered in an in-memory segment? [lucene]

["shubhamvishu (via GitHub)" <gi...@apache.org>]

shubhamvishu closed issue #15901: Are we undercounting RAM used by vectors buffered in an in-memory segment?
URL: https://github.com/apache/lucene/issues/15901


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org