You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sanjay dutt <sa...@yahoo.com.INVALID> on 2020/11/18 04:21:18 UTC
_version_ taking too much memory
Solr Collection setup
Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too much)Heap Space :- 12GB
So basically, above collection is having OutOfMemory issues. And upon inspection, I got to know that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field "_version_" is taking around 50 - 60% memory(5GB).
Now I am trying to understand How "_version_" internally using the field cache?
Around the same time, In Splunk logs I can see "Found MAX value from Terms for _version_ in index". and also few "firstSearcher" queries. Now I think upon core initialization "firstSearcher" triggered which then try to find the Max value for version using Field Cache but because of too many records it occupy lot of memory to carry out the task.
Can someone please validate Is what I am talking in above para is the problem or problem lies in somewhere else?
Best Regards,Sanjay
Re: _version_ taking too much memory
Posted by sanjay dutt <sa...@yahoo.com.INVALID>.
Thank you so much Shalin. Yes that would reduce the memory requirement at run time.But my next question, Is the "firstSearcher" doing this I mean due to which it tries to find the "Max value of version" and for that it it has to do the un-inversion.
Thanks and Regards,Sanjay Dutt
On Wednesday, November 18, 2020, 12:54:39 PM GMT+5:30, Shalin Shekhar Mangar <sh...@gmail.com> wrote:
You should change the _version_ field to be of type docValues and re-index
data. This way you don't have to pay the memory cost of un-inversion.
On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt
<sa...@yahoo.com.invalid> wrote:
> Solr Collection setup
> Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too
> much)Heap Space :- 12GB
> So basically, above collection is having OutOfMemory issues. And upon
> inspection, I got to know
> that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field
> "_version_" is taking around 50 - 60% memory(5GB).
> Now I am trying to understand How "_version_" internally using the field
> cache?
> Around the same time, In Splunk logs I can see "Found MAX value from Terms
> for _version_ in index". and also few "firstSearcher" queries. Now I think
> upon core initialization "firstSearcher" triggered which then try to find
> the Max value for version using Field Cache but because of too many records
> it occupy lot of memory to carry out the task.
> Can someone please validate Is what I am talking in above para is the
> problem or problem lies in somewhere else?
>
> Best Regards,Sanjay
>
--
Regards,
Shalin Shekhar Mangar.
Re: _version_ taking too much memory
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
You should change the _version_ field to be of type docValues and re-index
data. This way you don't have to pay the memory cost of un-inversion.
On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt
<sa...@yahoo.com.invalid> wrote:
> Solr Collection setup
> Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too
> much)Heap Space :- 12GB
> So basically, above collection is having OutOfMemory issues. And upon
> inspection, I got to know
> that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field
> "_version_" is taking around 50 - 60% memory(5GB).
> Now I am trying to understand How "_version_" internally using the field
> cache?
> Around the same time, In Splunk logs I can see "Found MAX value from Terms
> for _version_ in index". and also few "firstSearcher" queries. Now I think
> upon core initialization "firstSearcher" triggered which then try to find
> the Max value for version using Field Cache but because of too many records
> it occupy lot of memory to carry out the task.
> Can someone please validate Is what I am talking in above para is the
> problem or problem lies in somewhere else?
>
> Best Regards,Sanjay
>
--
Regards,
Shalin Shekhar Mangar.