You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "msokolov (via GitHub)" <gi...@apache.org> on 2024/03/05 18:08:24 UTC
[I] Create a read-only index that drops index files not needed for searching [lucene]
msokolov opened a new issue, #13158:
URL: https://github.com/apache/lucene/issues/13158
### Description
Now that we have vector quantization we face the possibility of writing an index that is 5 times bigger than is needed for searching. If the index is primarily vectors and they are quantized, we will save the full-precision vectors but they may not be required at all for searching. In an architecture where indexes are written on one set of hosts and replicated to another set of hosts for searching, it is wasteful to copy all of the full-precision vectors to the searcher nodes. But Lucene doesn't have any way of distinguishing. I wonder if we could create a "write read-only index" operation that would effectively clone the existing index, dropping any data required only for indexing, and mark the index as read-only so it could never be opened for writing. This might be useful in some way for version upgrades as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-2012935594
> Don't "deletes" require "writes"? Meaning, if enough docs get deleted in a segment, it will ultimately require to be merged, which then is a "write"?
The goal of segment-replication is to completely separate searching from writing, so in that world, no merging is done by searchers -- it happens upstream in a writer/indexer, or perhaps in a dedicated merger (we have both setups going on).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3443632462
A few questions:
* what would the API look like for stripping full-precision vectors and keeping only quantized ones? I guess it could be a codec option?
* what would happen if updates are made to the index configured with such a Codec? I think we'd ideally want to prevent that or somehow signal to the user that merges will fail. Ideally IndexWriter.add/updateDocument(s) would catch it, but I could also imagine a check in Directory (read-only Directory that cannot write any data would throw an exc), or possibly even a failure would get deferred until vector format attempts to merge and finds zero full precision vectors and nonzero quantized vectors? Otherwise it will silently delete all vector data...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-2012237415
Don't "deletes" require "writes"? Meaning, if enough docs get deleted in a segment, it will ultimately require to be merged, which then is a "write"?
For a quicker win in scalar quantization, it could be cheaper or easier to have a configured threshold where we throw away the floating point as we know the distribution of the values won't change significantly from that point on. Then on merges, if adjustments are required, we assume the cost of de-quantizing and re-quantizing. This can help relevancy more than you would expect, but obviously not as much as having access to the raw values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-2012195070
This is a cool idea @msokolov. It is wasteful to lug around those `float32` precision vectors out to the searchers in an NRT segment replication architecture. In practice, they would consume disk space on the searchers, and waste time copying them out, but since the OS would never load them at search time, their bytes would remain cold on disk and not put much pressure on OS free RAM? The OS would only cache the disk pages in the index that are actually needed at search time. It would be nice not to copy all that deadweight around ...
Probably the solution would have to be something like segment to segment? I.e. for each segment in the index, we would make a corresponding "read only" segment (stripped of the `float32` vectors). This way, as the normal index changes (gets new flushed/merged segments), we could also incrementally/NRT maintain the shadow read-only index.
I wonder if there are other things in a Lucene index today that are needed only during indexing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3648791542
> This saved us 80% of storage space on searchers and also reduced the downloading time from S3.
Just to clarify -- this is 80% smaller storage for just the vectors portion of the index. We (Amazon customer facing product search team -- I work with @Pulkitg64 and @msokolov) still have lots of other things in the Lucene index! Overall top-line reduction I think was ~10%, but that equates to PB (petabytes) of savings each day across the whole fleet! And as more and more vectors, with higher and higher dimensionality, are added to our indices, the vector portion of the index is a larger part, and these savings get bigger over time.
We can only do this because we fully rely on scalar quantized vectors for searching ... e.g. we never do 2nd phase reranking with full precision vectors. If a query wants to retrieve a vector as a return field, we re-hydrate the quantized form back to (lossy due to quantization round trip) full precision. And also because we have physical isolation of indexing and searching, using NRT segment replication (via S3 so we also get full, incremental backups on every commit point) to copy new segments on each commit from indexers to searchers.
As @Pulkitg64 described, our current solution is kinda hackity/messy because the Codec (and therefore `IndexWriter`, `SegmentInfos`, etc.) don't know about these files.
> For example, we could add support to write empty vector files directly from our codec and give users the flexibility to choose whether they want to use full-precision files or not.
+1, I like this approach.
Indexing would always write two sets of files (one with all the full precision vectors indexed, another with no vectors which will be tiny files -- just header and footer). Codec would own the inventory of these empty-full-precision files (adding them to `.files()`). And then they would be deleted at the right time since `IndexFileDeleter` would include them in ref counting (when that segment is merged away). And nothing would open them for reading, by default... they just "lurk"!
> A few questions:
Yeah this is the tricky part!
The one thing that needs to know it will open only for reading is the `KnnVectorsReader`, but we have no clean way to pass index-open-time parameters to Codecs I think? But you're right, we would also need the index to somehow store that it is a read-only index because something along the way dropped the full precision files? Not sure how to do that ... tricky part! It makes the empty-file-writing part seem easy lol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-2013042740
@msokolov AH, yes, for segment-replication, once the segments are built, I could see certain things being removed. I better understand the idea now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3429417165
Coming back to this issue:
### Summary
We tried to implement this idea in our closed-source implementation and got very good results, reduction in vector index size to one-fifth of the original size (80% reduction as mentioned in this issue).
### Background:
At Amazon, we have decoupled architecture where Lucene writers and searchers run on separate machines. Writers create the index and upload it to an S3 bucket, and searchers use the index after downloading it from the same S3 bucket.
Since full-precision float vectors are needed by writers for HNSW graph merging, we didn't modify anything there. However, once the index is read by searchers, assuming we don't need full-precision vectors anymore because only quantized vectors take part in vector scoring for HNSW searches, the full-precision vectors are just sitting idle on disk. So, in our closed-source implementation, we tried dropping full-precision vectors while downloading the checkpoint from S3 (explained in detail below).
### Implementation:
In our first/naive attempt, we simply tried to remove the full-precision vector files directly (vec and vem files), but this caused the codec to throw an IndexCorruptException.
Instead, here's what we did: While writing the index to the S3 bucket, Lucene writers uploaded additional empty full-precision vector files to S3.
Normally this is how these files look like
* vec:
```
* HEADER (Codec Magic, CodecName, Version, Segment ID, Segment-Suffix Length, Segment Suffix)
* Offset: To adjust start of vector data position (multiplies of Float Bytes i.e 4 Bytes)
* Data :
* Vectors: *Actual Float Vectors*
* -1 *(To mark end of the data)*
* FOOTER (FooterMagic, Checksum)
```
* vemf
```
* HEADER (Codec Magic, CodecName, Version, Segment ID, Segment-Suffix Length, Segment Suffix)
* Data
* Field Number
* Vector Encoding ordinal (Bytes/Float)
* Similarity Function ordinal (Cosine/Dot-Product etc)
* Start position of vectors in ```.vec``` file
* Total length of vectors
* Vector Dimension
* Total Vectors
* Ord to Doc Information
* -1 to mark end of field infos.
* FOOTER (FooterMagic, Checksum)
```
For our optimization, we created empty files like these:
* vec:
```
* HEADER (Codec Magic, CodecName, Version, Segment ID, Segment-Suffix Length, Segment Suffix)
* Offset: To adjust start of vector data position (multiplies of Float Bytes i.e 4 Bytes)
* Data : <<No vector data>>
* -1 *(To mark end of the data)*
* FOOTER (FooterMagic, Checksum)
```
* vemf
```
* HEADER (Codec Magic, CodecName, Version, Segment ID, Segment-Suffix Length, Segment Suffix)
* Data
* Field Number
* Vector Encoding ordinal (Bytes/Float) <<Same as original>>
* Similarity Function ordinal (Cosine/Dot-Product etc) <<Same as original>>
* Start position of vectors in ```.vec``` file <<Same as original>>
* Total length of vectors <<Zero in this case>>
* Vector Dimension <<Same as original>>
* Total Vectors <<Zero in this case>>>>
* Ord to Doc Information <<0 document information>>
* -1 to mark end of field infos.
* FOOTER (FooterMagic, Checksum)
```
On Lucene searchers, while downloading the index from S3, we skipped downloading the original full-precision vector files and instead downloaded only the empty full-precision vector files. This saved us 80% of storage space on searchers and also reduced the downloading time from S3.
### Next Steps:
Based on the above work, I wanted to know what the community thinks about this and whether we should implement this in the open-source Lucene repo as well. For example, we could add support to write empty vector files directly from our codec and give users the flexibility to choose whether they want to use full-precision files or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3773048389
Maybe we could separate the two changes? First, improve `KNNVectorsWriter` to always also write empty full precision vector files -- they are tiny. Then at least the hack (picking which file to replicate) is somewhat less dreadful? Then as follow-on we could work towards a clean API for Lucene users to express this at a clean API (no replication hacks).
> what would happen if updates are made to the index configured with such a Codec?
I think we could add some safety here (if the hack had bugs or so): `KnnVectorsWriter` merging could refuse to merge segments that are missing their full precision vectors. This way if a user replicates the index dropping full precision vectors, and then attempts to append to that index, they would at least see exceptions at merge time. Or maybe we could add a check on `IndexWriter` ctor that no segment X field is missing its full precision vectors?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3816890676
Hi,
how about a MergePolicy that strips the full vectors? As read-only indexes are mostly optimized to one segment anyways, maybe add a MergePolicy that can do this somehow when doing a forceMerge? It would of course be important that the force merge runs isolated and merges all segemnts in one go. If it would split the segments and merges multiple times, you would have loss due to merging 2 segments without full vectors. This is why I think that the lossy operation should be done as a post-processing step and now glued into IndexWriter or the general merging algorithms.
Still we need a way to tell the codec to disable full vectors, but this could be signalled with the type of merge (an extra "internal" IOContext flag added to the IOContext using our new IOContext-framework?)
To prevent merging of such an index at a later stage, we can of course add flags to the index file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "Pulkitg64 (via GitHub)" <gi...@apache.org>.
Pulkitg64 commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3815602440
Thanks @mikemccand and @msokolov for the responses and suggestions. TBH, I didn't have any good answers for the questions asked by @msokolov [here](https://github.com/apache/lucene/issues/13158#issuecomment-3443632462). But separating the change seems to be good idea to me. In [this PR](https://github.com/apache/lucene/pull/15630), I have added support for writing empty full-precision files. Please let me know if that approach makes sense to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3936085533
I think it's true that the reader side is where we need to do the work. The writer must write the full-precision vectors so it can read them for future updates/merges/etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3820323014
> As read-only indexes are mostly optimized to one segment anyways, maybe add a MergePolicy that can do this somehow when doing a forceMerge?
I think read-only isn't the right word -- you're thinking of the kind of indices you would burn & ship on CD so people could search the docs of installed software right :) Like "dead end index never again changed".
For this feature, the index is still writable / actively changing, but when you replicate it, the replica becomes read-only (replicas will never be promoted to primary and suddenly start writing into their copy).
Maybe a "read-only mirror/replica" is better term? Each new segment gets replicated to the read-only replica during NRT segment replication.
So this lossy process (discarding full precision vectors) is only done once, per segment, during replication, and IndexWriter never opens these replicated segments (only `IndexReader`).
Our hacky implementation today (during replication when we see a `.vec/q` file, we also write another file with 0 vectors instead) is quite messy because we are stuffing it into the index w/o Lucene's knowledge (cannot use Lucene's `incRef`/`decRef` for lifecycle, to delete them on time), we are recreating Codec-specific code (bad abstraction violation, since that's impl detail of the Codec), we have hacks to copy over the right segment GUID, match the Codec header/footer, write checksum, etc. -- badly copying all the stuff the Codec is already doing.
So the idea we iterated to here is to instead expect KNN quantizer (the quantizing `FlatVectorsWriter`) to also write another file, empty `.veq`. You would only use this empty `.veq` file if you want a "read only replica", then you would replicate this empty file instead of the full file. Since the Codec then owns this file, it's properly tracked in `.files()`, deleted when segment is deleted, etc.
I *think* it should be a tiny change (compared to internal hack we have so far...), but I haven't looked at #15630 yet to see...
This design is not perfect -- most Lucene users won't use these empty `.veq` files (though they are tiny, and never opened unless you make read-only replica).
But the impact is _massive_ for our (Amazon's Customer product search) usage -- 5X reduction in index size / replication bandwidth / storage / time for vector-heavy indices (because for `int8` we are storing both `float32` and `int8` (five bytes) per dimension, and with this fix it's one byte).
So I hope we can find a clean / simple way to bring this option to all Lucene users using NRT segment replication ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Create a read-only index that drops index files not needed for searching [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #13158:
URL: https://github.com/apache/lucene/issues/13158#issuecomment-3820374506
@mikemccand Ah, I see the challenges (don't fully understand them yet myself)
I left a comment on https://github.com/apache/lucene/pull/15630 that might be out of ignorance, but, that PR just looks like a formalization of the "hacky process".
Ah man, it feels like the reader should be able to just access the files it needs and never reference the raw vector file and the directory would then be "lazy" and just never fully down load it. This of course, doesn't handle CFS, and likely has a plethora of edges I haven't thought of...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org