You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Tanmay Goel <go...@gmail.com> on 2024/10/29 19:08:38 UTC
How to find RAM/disk usage of each vector field
Hi all
I recently joined the Lucene team at Amazon and this is my first time
working with Lucene so any help will be appreciated.
One of my first tasks is to *add a metric in production to track the RAM /
disk usage of vector fields*. We want to use this metric to decide when to
scale our deployments.
One of the ideas to get this data was to split the index files such that we
have separate files for each field and prefix filenames with the
field name. We could then analyze the index files and figure out how many
bytes are used for each field. However, this idea is called out as a bad
practice in Lucene docs (
https://github.com/apache/lucene/blob/main/dev-docs/file-formats.md#dont-use-too-many-files
)
Is there any other way to find out how many bytes are being used by vector
fields?
Thanks!
Tanmay
Re: How to find RAM/disk usage of each vector field
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Nov 5, 2024 at 7:31 PM Patrick Zhai <zh...@gmail.com> wrote:
I wouldn't call this a good way, but as the last resort you can parse the
> metadata files yourself, as it is not so hard to parse (yet)
Yeah ... the Lucene codec itself knows precisely how much disk is used for
each field, and indeed stores it simply in its metadata. And it's
incredibly fast to peek into that metadata to get the per-field metrics.
We will likely take this approach (on top of Lucene) but it is clearly an
abstraction violation, is brittle to future Codec changes, not supported,
etc.
We've taken the same brittle approach (at Amazon product search) to track
per-field disk usage of terms dictionary and postings (inverted lexical
index) for similar reasons (so many teams indexing so many fields with so
many words!).
I would think many multi-tenant users of Lucene would want some resource
tracking along these lines... but "officially" supporting this in Lucene's
Codec APIs would be an added dev burden.
As for RAM usage, OnHeapHnswGraph actually implements the Accountable API
> <
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/hnsw/OnHeapHnswGraph.java#L279C15-L279C27
> >,
> and HnswGraphBuilder also have InfoStream passed in
> <
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java#L199
> >
> so
> I think
> it's ok and reasonable to report the RAM usage at the end of graph build
> maybe? Tho this won't include the off heap vector sizes but that one can be
> estimated easily I think?
>
I think the OnHeapHnswGraph is used only during indexing? But +1 to have
the infoStream print the RAM size of that graph during indexing if it
doesn't already ... Tanmay maybe open a spinoff issue for this small
improvement?
Thanks Patrick.
Mike McCandless
http://blog.mikemccandless.com
Re: How to find RAM/disk usage of each vector field
Posted by Patrick Zhai <zh...@gmail.com>.
I wouldn't call this a good way, but as the last resort you can parse the
metadata files yourself, as it is not so hard to parse (yet), the logics
are in:
Lucene99HnswVectorsFormat.java
Lucene99FlatVectorsFormat.java
The risk for sure is that whenever the format is changed the parsing logic
will need to be changed accordingly, and I don't think we will/can
guarantee any consistency on the format nor do we want to maintain such a
disk usage estimation tool in the lucene codebase
As for RAM usage, OnHeapHnswGraph actually implements the Accountable API
<https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/hnsw/OnHeapHnswGraph.java#L279C15-L279C27>,
and HnswGraphBuilder also have InfoStream passed in
<https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java#L199>
so
I think
it's ok and reasonable to report the RAM usage at the end of graph build
maybe? Tho this won't include the off heap vector sizes but that one can be
estimated easily I think?
On Tue, Nov 5, 2024 at 2:16 PM Adrien Grand <jp...@gmail.com> wrote:
> I cannot think of good ways to do this. Why is it important to break down
> per field as opposed to scaling based on the total volume of vector data?
>
> On Tue, Nov 5, 2024 at 10:58 PM Tanmay Goel <go...@gmail.com> wrote:
>
> > Hi Rui
> >
> > Thanks for your response and the snippet that you shared is great but not
> > exactly what I was looking for. With this snippet we are able to find the
> > total size of the .vec files, but I want to see inside the .vec files and
> > try to compute a map of vector_field_name to the number of bytes on disk.
> >
> > Thanks
> > Tanmay
> >
> > On Wed, 30 Oct 2024 at 13:30, Rui Wu <ru...@mongodb.com.invalid> wrote:
> >
> > > Hi Tanmay,
> > >
> > > Are you bothered by the .vec files hidden within the compound files? If
> > > yes, I have a snippet that can sum up the .vec files inside and outside
> > > compound files.
> > > https://gist.github.com/wurui90/28de20d46079108d7ae5ed181ba939d4
> > >
> > > On Tue, Oct 29, 2024 at 12:08 PM Tanmay Goel <go...@gmail.com>
> > wrote:
> > >
> > > > Hi all
> > > >
> > > > I recently joined the Lucene team at Amazon and this is my first time
> > > > working with Lucene so any help will be appreciated.
> > > >
> > > > One of my first tasks is to *add a metric in production to track the
> > RAM
> > > /
> > > > disk usage of vector fields*. We want to use this metric to decide
> when
> > > to
> > > > scale our deployments.
> > > >
> > > > One of the ideas to get this data was to split the index files such
> > that
> > > we
> > > > have separate files for each field and prefix filenames with the
> > > > field name. We could then analyze the index files and figure out how
> > many
> > > > bytes are used for each field. However, this idea is called out as a
> > bad
> > > > practice in Lucene docs (
> > > >
> > > >
> > >
> >
> https://github.com/apache/lucene/blob/main/dev-docs/file-formats.md#dont-use-too-many-files
> > > > )
> > > >
> > > > Is there any other way to find out how many bytes are being used by
> > > vector
> > > > fields?
> > > >
> > > > Thanks!
> > > >
> > > > Tanmay
> > > >
> > >
> >
>
>
> --
> Adrien
>
Re: How to find RAM/disk usage of each vector field
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Nov 5, 2024 at 5:17 PM Adrien Grand <jp...@gmail.com> wrote
Why is it important to break down per field as opposed to scaling based on
> the total volume of vector data?
>
It's really for internal planning purposes / service telemetry ... at
Amazon product search team (where I also work w/ Tanmay -- hi Tanmay!) we
have a number of teams using our Lucene search service to experiment with
KNN search, varying the number of dimensions, whether quantization is in
use, which ML model, etc. These fields come and go, sometimes without our
(low level infrastructure) service knowing ahead of time how these fields
are changing. So we would ideally have an efficient way to break out
per-field KNN disk and "ideal hot RAM" online (in our production service)
instead of offline / inefficiently like rewriting the whole index into
separate files (Robert's cool DiskUsage tool).
It's tricky with KNN and features like scalar quantization (
https://www.elastic.co/search-labs/blog/scalar-quantization-in-lucene) and
soon RabitQ (https://github.com/apache/lucene/pull/13651) because the
on-disk form (which retains full float32 precision vectors) is different
from what searching really uses (the quantized byte-per-dimension form).
So the disk consumed by each field is larger than the amount of effective
"hot RAM" you might need.
Mike McCandless
http://blog.mikemccandless.com
Re: How to find RAM/disk usage of each vector field
Posted by Adrien Grand <jp...@gmail.com>.
I cannot think of good ways to do this. Why is it important to break down
per field as opposed to scaling based on the total volume of vector data?
On Tue, Nov 5, 2024 at 10:58 PM Tanmay Goel <go...@gmail.com> wrote:
> Hi Rui
>
> Thanks for your response and the snippet that you shared is great but not
> exactly what I was looking for. With this snippet we are able to find the
> total size of the .vec files, but I want to see inside the .vec files and
> try to compute a map of vector_field_name to the number of bytes on disk.
>
> Thanks
> Tanmay
>
> On Wed, 30 Oct 2024 at 13:30, Rui Wu <ru...@mongodb.com.invalid> wrote:
>
> > Hi Tanmay,
> >
> > Are you bothered by the .vec files hidden within the compound files? If
> > yes, I have a snippet that can sum up the .vec files inside and outside
> > compound files.
> > https://gist.github.com/wurui90/28de20d46079108d7ae5ed181ba939d4
> >
> > On Tue, Oct 29, 2024 at 12:08 PM Tanmay Goel <go...@gmail.com>
> wrote:
> >
> > > Hi all
> > >
> > > I recently joined the Lucene team at Amazon and this is my first time
> > > working with Lucene so any help will be appreciated.
> > >
> > > One of my first tasks is to *add a metric in production to track the
> RAM
> > /
> > > disk usage of vector fields*. We want to use this metric to decide when
> > to
> > > scale our deployments.
> > >
> > > One of the ideas to get this data was to split the index files such
> that
> > we
> > > have separate files for each field and prefix filenames with the
> > > field name. We could then analyze the index files and figure out how
> many
> > > bytes are used for each field. However, this idea is called out as a
> bad
> > > practice in Lucene docs (
> > >
> > >
> >
> https://github.com/apache/lucene/blob/main/dev-docs/file-formats.md#dont-use-too-many-files
> > > )
> > >
> > > Is there any other way to find out how many bytes are being used by
> > vector
> > > fields?
> > >
> > > Thanks!
> > >
> > > Tanmay
> > >
> >
>
--
Adrien
Re: How to find RAM/disk usage of each vector field
Posted by Tanmay Goel <go...@gmail.com>.
Hi Rui
Thanks for your response and the snippet that you shared is great but not
exactly what I was looking for. With this snippet we are able to find the
total size of the .vec files, but I want to see inside the .vec files and
try to compute a map of vector_field_name to the number of bytes on disk.
Thanks
Tanmay
On Wed, 30 Oct 2024 at 13:30, Rui Wu <ru...@mongodb.com.invalid> wrote:
> Hi Tanmay,
>
> Are you bothered by the .vec files hidden within the compound files? If
> yes, I have a snippet that can sum up the .vec files inside and outside
> compound files.
> https://gist.github.com/wurui90/28de20d46079108d7ae5ed181ba939d4
>
> On Tue, Oct 29, 2024 at 12:08 PM Tanmay Goel <go...@gmail.com> wrote:
>
> > Hi all
> >
> > I recently joined the Lucene team at Amazon and this is my first time
> > working with Lucene so any help will be appreciated.
> >
> > One of my first tasks is to *add a metric in production to track the RAM
> /
> > disk usage of vector fields*. We want to use this metric to decide when
> to
> > scale our deployments.
> >
> > One of the ideas to get this data was to split the index files such that
> we
> > have separate files for each field and prefix filenames with the
> > field name. We could then analyze the index files and figure out how many
> > bytes are used for each field. However, this idea is called out as a bad
> > practice in Lucene docs (
> >
> >
> https://github.com/apache/lucene/blob/main/dev-docs/file-formats.md#dont-use-too-many-files
> > )
> >
> > Is there any other way to find out how many bytes are being used by
> vector
> > fields?
> >
> > Thanks!
> >
> > Tanmay
> >
>
Re: How to find RAM/disk usage of each vector field
Posted by Rui Wu <ru...@mongodb.com.INVALID>.
Hi Tanmay,
Are you bothered by the .vec files hidden within the compound files? If
yes, I have a snippet that can sum up the .vec files inside and outside
compound files.
https://gist.github.com/wurui90/28de20d46079108d7ae5ed181ba939d4
On Tue, Oct 29, 2024 at 12:08 PM Tanmay Goel <go...@gmail.com> wrote:
> Hi all
>
> I recently joined the Lucene team at Amazon and this is my first time
> working with Lucene so any help will be appreciated.
>
> One of my first tasks is to *add a metric in production to track the RAM /
> disk usage of vector fields*. We want to use this metric to decide when to
> scale our deployments.
>
> One of the ideas to get this data was to split the index files such that we
> have separate files for each field and prefix filenames with the
> field name. We could then analyze the index files and figure out how many
> bytes are used for each field. However, this idea is called out as a bad
> practice in Lucene docs (
>
> https://github.com/apache/lucene/blob/main/dev-docs/file-formats.md#dont-use-too-many-files
> )
>
> Is there any other way to find out how many bytes are being used by vector
> fields?
>
> Thanks!
>
> Tanmay
>