You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Heinz Hölzer <he...@googlemail.com.INVALID> on 2023/02/07 11:33:53 UTC
Memory cost of unused indexed and docValues properties
Hi,
We are currently optimizing and cleaning up our complex Solr Schema and we
are wondering if the corresponding data structures of a field with
indexed="true" and docValues="true" (inverted index, column-oriented
mapping) are loaded from disk into memory when both features are never used
or accessed? In other words, can we decrease our memory usage by setting
indexed and docValues to false even if those features are never used at
runtime?
Thx and best regards,
Heinz
Re: Memory cost of unused indexed and docValues properties
Posted by Mikhail Khludnev <mk...@apache.org>.
Pls check inside below.
On Fri, Feb 10, 2023 at 9:21 AM Heinz Hölzer
<he...@googlemail.com.invalid> wrote:
> Hi Mikhail,
>
> Thank a lot for your answer, this means we will only pay a small heap price
> for both docValues and indexed, right?
Exactly. Perhaps this idea expressed
https://www.youtube.com/watch?v=T5RmMNDR5XI
Some time ago Lucene had an API to report heap used by internal
components... until https://issues.apache.org/jira/browse/LUCENE-9387
> Thx for your `rawSize=true` finding,
> this will be helpful to inspect disk usage.
>
> Best,
> Heinz
>
> Am Di., 7. Feb. 2023 um 21:21 Uhr schrieb Mikhail Khludnev <
> mkhl@apache.org
> >:
>
> > Hello, Heinz.
> > These data structures reside on disk and only small excerpts ie head is
> > kept in heap. So, dropping them you should get some gain in heap, but I
> > don't think it's a lot. Indexing footprint should be reduced as well.
> > Regarding heap size. I've found a pretty cool param rawSize=true in
> >
> >
> http://localhost:8983/solr/admin/collections?action=COLSTATUS&collection=gettingstarted&coreInfo=true&segments=true&fieldInfo=true&sizeInfo=true&rawSize=true
> > which estimates field size, but it's a disk size, not heap usage. It
> seems
> > to me that heap size estimation is abandoned.
> >
> > On Tue, Feb 7, 2023 at 2:34 PM Heinz Hölzer
> > <he...@googlemail.com.invalid> wrote:
> >
> > > Hi,
> > >
> > > We are currently optimizing and cleaning up our complex Solr Schema and
> > we
> > > are wondering if the corresponding data structures of a field with
> > > indexed="true" and docValues="true" (inverted index, column-oriented
> > > mapping) are loaded from disk into memory when both features are never
> > used
> > > or accessed? In other words, can we decrease our memory usage by
> setting
> > > indexed and docValues to false even if those features are never used at
> > > runtime?
> > >
> > > Thx and best regards,
> > > Heinz
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > https://t.me/MUST_SEARCH
> > A caveat: Cyrillic!
> >
>
--
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!
Re: Memory cost of unused indexed and docValues properties
Posted by Heinz Hölzer <he...@googlemail.com.INVALID>.
Hi Mikhail,
Thank a lot for your answer, this means we will only pay a small heap price
for both docValues and indexed, right? Thx for your `rawSize=true` finding,
this will be helpful to inspect disk usage.
Best,
Heinz
Am Di., 7. Feb. 2023 um 21:21 Uhr schrieb Mikhail Khludnev <mkhl@apache.org
>:
> Hello, Heinz.
> These data structures reside on disk and only small excerpts ie head is
> kept in heap. So, dropping them you should get some gain in heap, but I
> don't think it's a lot. Indexing footprint should be reduced as well.
> Regarding heap size. I've found a pretty cool param rawSize=true in
>
> http://localhost:8983/solr/admin/collections?action=COLSTATUS&collection=gettingstarted&coreInfo=true&segments=true&fieldInfo=true&sizeInfo=true&rawSize=true
> which estimates field size, but it's a disk size, not heap usage. It seems
> to me that heap size estimation is abandoned.
>
> On Tue, Feb 7, 2023 at 2:34 PM Heinz Hölzer
> <he...@googlemail.com.invalid> wrote:
>
> > Hi,
> >
> > We are currently optimizing and cleaning up our complex Solr Schema and
> we
> > are wondering if the corresponding data structures of a field with
> > indexed="true" and docValues="true" (inverted index, column-oriented
> > mapping) are loaded from disk into memory when both features are never
> used
> > or accessed? In other words, can we decrease our memory usage by setting
> > indexed and docValues to false even if those features are never used at
> > runtime?
> >
> > Thx and best regards,
> > Heinz
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> https://t.me/MUST_SEARCH
> A caveat: Cyrillic!
>
Re: Memory cost of unused indexed and docValues properties
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, Heinz.
These data structures reside on disk and only small excerpts ie head is
kept in heap. So, dropping them you should get some gain in heap, but I
don't think it's a lot. Indexing footprint should be reduced as well.
Regarding heap size. I've found a pretty cool param rawSize=true in
http://localhost:8983/solr/admin/collections?action=COLSTATUS&collection=gettingstarted&coreInfo=true&segments=true&fieldInfo=true&sizeInfo=true&rawSize=true
which estimates field size, but it's a disk size, not heap usage. It seems
to me that heap size estimation is abandoned.
On Tue, Feb 7, 2023 at 2:34 PM Heinz Hölzer
<he...@googlemail.com.invalid> wrote:
> Hi,
>
> We are currently optimizing and cleaning up our complex Solr Schema and we
> are wondering if the corresponding data structures of a field with
> indexed="true" and docValues="true" (inverted index, column-oriented
> mapping) are loaded from disk into memory when both features are never used
> or accessed? In other words, can we decrease our memory usage by setting
> indexed and docValues to false even if those features are never used at
> runtime?
>
> Thx and best regards,
> Heinz
>
--
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!