You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Satya Nand <sa...@indiamart.com.INVALID> on 2021/08/09 05:43:29 UTC
OutofMemory Error in solr 6.5
Hi,
We are facing a strange issue in our solr system. Most of the days it keeps
running fine but once or twice in a month, we face OutofMemory on solr
servers.
We are using Leader-Follower architecture, one Leader and 4 followers.
Strangely we get OutofMemory error on all follower servers.
Before the OutOfMemory this exception is found on all servers.
Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall
search-central-prd-solr-temp1
ERROR: null:java.lang.NullPointerException search-central-prd-solr-temp1
at
org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
search-central-prd-solr-temp1
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
search-central-prd-solr-temp1
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
search-central-prd-solr-temp1
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
search-central-prd-solr-temp1
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at org.eclipse.jetty.server.Server.handle(Server.java:534)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
search-central-prd-solr-temp1
*Configuration*
Index size- 95 GB
Heap 30 GB
Ram-96 GB
Polling Interval - 3 Hours
Caching-
<filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
autowarmCount="100" /> <queryResultCache class="solr.LRUCache" size="30000"
initialSize="1000" autowarmCount="100" /> <documentCache class=
"solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
--
Re: OutofMemory Error in solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/12/2021 12:02 AM, Satya Nand wrote:
> Does this alternate format use different data structures to store the
> document ids for filters with low document count, Other than the bitmap?
>
> means the size constraint(filter cache size) would apply only on bitmap or
> this alternate structure too or their sum?
Yes, the alternate format is just a list of IDs, each of which is a
signed 32-bit integer (four bytes). So it will be far smaller than a
bitmap, as it is only used when the hitcount for the filter is low.
But since there's no way to know in advance whether a particular filter
meets the threshold to use the alternate format, it's best to prepare
for every filterCache entry to be the larger bitmap format. If you take
steps to split your filters into multiple fq parameters to help
performance, chances are that almost all of the cache entries WILL be
the bitmap format.
Thanks,
Shawn
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Thanks, Shawn,
This makes sense. Filter queries with high hit counts could be the trigger
for out-of-memory, That's why it is so infrequent.
We will try to relook filter queries and further try reducing filter cache
size.
one question though,
> There is an alternate format for filterCache entries, that just lists
> the IDs of the matching documents. This only gets used when the
> hitcount for the filter is low.
Does this alternate format use different data structures to store the
document ids for filters with low document count, Other than the bitmap?
means the size constraint(filter cache size) would apply only on bitmap or
this alternate structure too or their sum?
On Wed, 11 Aug, 2021, 6:50 pm Shawn Heisey, <ap...@elyograg.org> wrote:
> On 8/11/2021 6:04 AM, Satya Nand wrote:
> > *Filter cache stats:*
> >
> https://drive.google.com/file/d/19MHEzi9m3KS4s-M86BKFiwmnGkMh3DGx/view?usp=sharing
>
>
> This shows the current size as 3912, almost full.
>
> There is an alternate format for filterCache entries, that just lists
> the IDs of the matching documents. This only gets used when the
> hitcount for the filter is low. I do not know what threshold it uses to
> decide that the hitcount is low enough to use the alternate format, and
> I do not know where in the code to look for the answer. This is
> probably why you can have 3912 entries in the cache without blowing the
> heap.
>
> I bet that when the heap gets blown, the filter queries Solr receives
> are such that they cannot use the alternate format, and thus require the
> full 12.7 million bytes. Get enough of those, and you're going to need
> more heap than 30GB. I bet that if you set the heap to 31G, the OOMEs
> would occur a little less frequently. Note that if you set the heap to
> 32G, you actually have less memory available than if you set it to 31G
> -- At 32GB, Java must switch from 32 bit pointers to 64 bit pointers.
> Solr creates a LOT of objects on the heap, so that difference adds up.
>
> Discussion item for those with an interest in the low-level code: What
> kind of performance impact would it cause to use a filter bitmap
> compressed with run-length encoding? Would that happen at the Lucene
> level rather than the Solr level?
>
> To fully solve this issue, you may need to re-engineer your queries so
> that fq values are highly reusable, and non-reusable filters are added
> to the main query. Then you would not need a very large cache to obtain
> a good hit ratio.
>
> Thanks,
> Shawn
>
>
--
Re: OutofMemory Error in solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/11/2021 6:04 AM, Satya Nand wrote:
> *Filter cache stats:*
> https://drive.google.com/file/d/19MHEzi9m3KS4s-M86BKFiwmnGkMh3DGx/view?usp=sharing
This shows the current size as 3912, almost full.
There is an alternate format for filterCache entries, that just lists
the IDs of the matching documents. This only gets used when the
hitcount for the filter is low. I do not know what threshold it uses to
decide that the hitcount is low enough to use the alternate format, and
I do not know where in the code to look for the answer. This is
probably why you can have 3912 entries in the cache without blowing the
heap.
I bet that when the heap gets blown, the filter queries Solr receives
are such that they cannot use the alternate format, and thus require the
full 12.7 million bytes. Get enough of those, and you're going to need
more heap than 30GB. I bet that if you set the heap to 31G, the OOMEs
would occur a little less frequently. Note that if you set the heap to
32G, you actually have less memory available than if you set it to 31G
-- At 32GB, Java must switch from 32 bit pointers to 64 bit pointers.
Solr creates a LOT of objects on the heap, so that difference adds up.
Discussion item for those with an interest in the low-level code: What
kind of performance impact would it cause to use a filter bitmap
compressed with run-length encoding? Would that happen at the Lucene
level rather than the Solr level?
To fully solve this issue, you may need to re-engineer your queries so
that fq values are highly reusable, and non-reusable filters are added
to the main query. Then you would not need a very large cache to obtain
a good hit ratio.
Thanks,
Shawn
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Shawn,
Please find the images.
*Filter cache stats:*
https://drive.google.com/file/d/19MHEzi9m3KS4s-M86BKFiwmnGkMh3DGx/view?usp=sharing
*Heap stats*
https://drive.google.com/file/d/1Q62ea-nFh9UjbcVcBJ39AECWym6nk2Yg/view?usp=sharing
I'm curious whether the 101 million document count is for one shard
> replica or for the whole collection. How many documents are in all the
> shard replicas handled by one Solr instance?
We are not using solr cloud. We are using standalone solr with
Master-slave architecture. 101 million documents are in one core.
On Wed, Aug 11, 2021 at 5:20 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/10/2021 11:17 PM, Satya Nand wrote:
> > Thanks for explaining it so well. We will work on reducing the filter
> > cache size and auto warm count.
> >
> > Though I have one question.
> >
> > If your configured 4000 entry filterCache were to actually fill up,
> it
> > would require nearly 51 billion bytes, and that's just for the one
> > core
> > with 101 million documents. This is much larger than the 30GB
> > heap you
> > have specified ... I am betting that the filterCache is the reason
> > you're hitting OOME.
> >
> >
> > As you can see from the below screenshots the filter cache is almost
> > full and the heap is approx 18-20 GB. I think this means heap is not
> > actually taking 51 GB of space. Otherwise, the issue would have been
> > very frequent if the full cache had been taking ~50 GB of space. I
> > also believed the solr uses some compressed data structures to
> > accumulate its cache, That' how it is able to store the cache in less
> > memory. Isn't it?
> >
> > Also, the issue is not very frequent. It comes once or twice a month,
> > Where all follower servers stop working at the same time due to
> > OutOfMemory error.
>
> We can't see any of the images. The mailing list software stripped
> them. Most attachments do not come through -- you'll need to find a
> file sharing website and give us links. Dropbox is a good choice, and
> there are others.
>
> The cache may not be getting full, but each entry is over 12 megabytes
> in size, so it will not need to be full to cause problems. It does not
> get compressed. Solr (actually Lucene) does use compression in the
> index file formats. It would be possible to compress the bitmap for a
> filterCache entry, but that would slow things down when there is a cache
> hit. I have no idea how much it would slow things down.
>
> The cache warming probably isn't the problem. That's only going to
> (temporarily) add 100 new entries to a new cache, then the old cache
> will be gone. If the filterCache is indeed the major memory usage, it's
> probably queries that cause it to get large.
>
> I'm curious whether the 101 million document count is for one shard
> replica or for the whole collection. How many documents are in all the
> shard replicas handled by one Solr instance?
>
> Thanks,
> Shawn
>
>
>
--
Re: OutofMemory Error in solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/10/2021 11:17 PM, Satya Nand wrote:
> Thanks for explaining it so well. We will work on reducing the filter
> cache size and auto warm count.
>
> Though I have one question.
>
> If your configured 4000 entry filterCache were to actually fill up, it
> would require nearly 51 billion bytes, and that's just for the one
> core
> with 101 million documents. This is much larger than the 30GB
> heap you
> have specified ... I am betting that the filterCache is the reason
> you're hitting OOME.
>
>
> As you can see from the below screenshots the filter cache is almost
> full and the heap is approx 18-20 GB. I think this means heap is not
> actually taking 51 GB of space. Otherwise, the issue would have been
> very frequent if the full cache had been taking ~50 GB of space. I
> also believed the solr uses some compressed data structures to
> accumulate its cache, That' how it is able to store the cache in less
> memory. Isn't it?
>
> Also, the issue is not very frequent. It comes once or twice a month,
> Where all follower servers stop working at the same time due to
> OutOfMemory error.
We can't see any of the images. The mailing list software stripped
them. Most attachments do not come through -- you'll need to find a
file sharing website and give us links. Dropbox is a good choice, and
there are others.
The cache may not be getting full, but each entry is over 12 megabytes
in size, so it will not need to be full to cause problems. It does not
get compressed. Solr (actually Lucene) does use compression in the
index file formats. It would be possible to compress the bitmap for a
filterCache entry, but that would slow things down when there is a cache
hit. I have no idea how much it would slow things down.
The cache warming probably isn't the problem. That's only going to
(temporarily) add 100 new entries to a new cache, then the old cache
will be gone. If the filterCache is indeed the major memory usage, it's
probably queries that cause it to get large.
I'm curious whether the 101 million document count is for one shard
replica or for the whole collection. How many documents are in all the
shard replicas handled by one Solr instance?
Thanks,
Shawn
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Shawn,
Thanks for explaining it so well. We will work on reducing the filter cache
size and auto warm count.
Though I have one question.
If your configured 4000 entry filterCache were to actually fill up, it
> would require nearly 51 billion bytes, and that's just for the one core
> with 101 million documents. This is much larger than the 30GB heap you
> have specified ... I am betting that the filterCache is the reason
> you're hitting OOME.
As you can see from the below screenshots the filter cache is almost full
and the heap is approx 18-20 GB. I think this means heap is not actually
taking 51 GB of space. Otherwise, the issue would have been very frequent
if the full cache had been taking ~50 GB of space. I also believed the solr
uses some compressed data structures to accumulate its cache, That' how it
is able to store the cache in less memory. Isn't it?
Also, the issue is not very frequent. It comes once or twice a month, Where
all follower servers stop working at the same time due to OutOfMemory error.
*Filter Cache statics as of 10:08 IST*
[image: image.png]
*Heap Usages*
[image: image.png]
-
On Wed, Aug 11, 2021 at 4:12 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/10/2021 1:06 AM, Satya Nand wrote:
> > Document count is 101893353.
>
> The OOME exception confirms that we are dealing with heap memory. That
> means we won't have to look into the other resource types that can cause
> OOME.
>
> With that document count, each filterCache entry is 12736670 bytes, plus
> some small number of bytes for java object overhead. That's 12.7
> million bytes.
>
> If your configured 4000 entry filterCache were to actually fill up, it
> would require nearly 51 billion bytes, and that's just for the one core
> with 101 million documents. This is much larger than the 30GB heap you
> have specified ... I am betting that the filterCache is the reason
> you're hitting OOME.
>
> You need to dramatically reduce the size of your filterCache. Start
> with 256 and see what that gets you. Solr ships with a size of 512.
> Also, see what you can do about making it so that there is a lot of
> re-use possible with queries that you put in the fq parameter. It's
> better to have several fq parameters rather than one parameter with a
> lot of AND clauses -- much more chance of filter re-use.
>
> I notice that you have autowarmCount set to 100 on two caches. (The
> autowarmCount on the documentCache, which you have set to 512, won't be
> used -- that cache cannot be warmed directly. It is indirectly warmed
> when the other caches are warmed.) This means that every time you issue
> a commit that opens a new searcher, Solr will execute up to 200 queries
> as part of the cache warming. This can make the warming take a VERY
> long time. Consider reducing autowarmCount. It's not causing your OOME
> problems, but it might be making commits take a very long time.
>
> Thanks,
> Shawn
>
--
Re: OutofMemory Error in solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/10/2021 1:06 AM, Satya Nand wrote:
> Document count is 101893353.
The OOME exception confirms that we are dealing with heap memory. That
means we won't have to look into the other resource types that can cause
OOME.
With that document count, each filterCache entry is 12736670 bytes, plus
some small number of bytes for java object overhead. That's 12.7
million bytes.
If your configured 4000 entry filterCache were to actually fill up, it
would require nearly 51 billion bytes, and that's just for the one core
with 101 million documents. This is much larger than the 30GB heap you
have specified ... I am betting that the filterCache is the reason
you're hitting OOME.
You need to dramatically reduce the size of your filterCache. Start
with 256 and see what that gets you. Solr ships with a size of 512.
Also, see what you can do about making it so that there is a lot of
re-use possible with queries that you put in the fq parameter. It's
better to have several fq parameters rather than one parameter with a
lot of AND clauses -- much more chance of filter re-use.
I notice that you have autowarmCount set to 100 on two caches. (The
autowarmCount on the documentCache, which you have set to 512, won't be
used -- that cache cannot be warmed directly. It is indirectly warmed
when the other caches are warmed.) This means that every time you issue
a commit that opens a new searcher, Solr will execute up to 200 queries
as part of the cache warming. This can make the warming take a VERY
long time. Consider reducing autowarmCount. It's not causing your OOME
problems, but it might be making commits take a very long time.
Thanks,
Shawn
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Dominique,
Thanks, But I still have one confusion, Please help me with it.
Pretty sure the issue is caused by caches size at new searcher warmup time.
We use leader-follower architecture with a replication interval of 3 hours.
This means every 3 hours we get a commit and the searcher warms up. Right?
We have frequent indexing, It isn't possible that we don't get any document
indexed in 3 hours period.
So the new searcher warms up 7-8 times a day. If the issue was due to a new
searcher warm-up, We would have got the issue many times a day. But we get
this OutOfMemory Issue once or twice a month or sometimes once in two
months, so some combination of events must be triggering it.
Further, In solrConfig we have
<maxWarmingSearchers>2</maxWarmingSearchers>
does this mean that at some time up to 3 searchers(1 active and 2 still
warming up their caches) might be holding to their caches and this
resulting in the issue? Sould we reduce this to 1 ? what impact would it
have if we reduced it ?
On Wed, Aug 11, 2021 at 2:15 AM Dominique Bejean <do...@eolya.fr>
wrote:
> Pretty sure the issue is caused by caches size at new searcher warmup time.
>
> Dominique
>
>
> Le mar. 10 août 2021 à 09:07, Satya Nand <sa...@indiamart.com> a
> écrit :
>
>> Hi Dominique,
>>
>> You don't provide information about the number of documents. Anyway, all
>>> your cache size and mostly initial size are big. Cache are stored in JVM
>>> heap.
>>
>> Document count is 101893353.
>>
>> About cache size, most is not always better. Did you make some performance
>>> benchmarks in order to set these values ?
>>
>> We increased cache size in the hope to reduce some response time, We
>> heavily use group queries with 7-8 boost factors. The average response time
>> on this document set is 136 ms. We receive approx 65 requests/second in
>> peak hours. The replication interval is 3 hours.
>>
>> The most strange thing about is that system keeps running for days
>> without any issue, So I believe cache size should not be an issue. If the
>> cache size had been the culprit, the issue would have been frequent. isn't
>> it?
>>
>>
>>
>> On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean <
>> dominique.bejean@eolya.fr> wrote:
>>
>>> Hi,
>>>
>>> You don't provide information about the number of documents. Anyway, all
>>> your cache size and mostly initial size are big. Cache are stored in JVM
>>> heap.
>>>
>>> About cache size, most is not always better. Did you make some
>>> performance
>>> benchmarks in order to set these values ?
>>>
>>> Try with the default values, after a few hours check cumulative caches
>>> statistics in order to decide if you need to increase their sizes. The
>>> objective is not to have cumulative_hitratio to 100%. There isn't ideal
>>> value as it is really related to your datas, to the user's queries, to
>>> how
>>> you build your queries ... but 70% is a good value. At some point
>>> increasing the size again and again won't increase cumulative_hitratio a
>>> lot as it is a logarithmic curve.
>>>
>>> Check also the heap usage also with your JVM GC logs and a tool like
>>> gceasy.io
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>>
>>>
>>>
>>> Le lun. 9 août 2021 à 07:44, Satya Nand <satya.nand@indiamart.com
>>> .invalid>
>>> a écrit :
>>>
>>> > Hi,
>>> > We are facing a strange issue in our solr system. Most of the days it
>>> keeps
>>> > running fine but once or twice in a month, we face OutofMemory on solr
>>> > servers.
>>> >
>>> > We are using Leader-Follower architecture, one Leader and 4 followers.
>>> > Strangely we get OutofMemory error on all follower servers.
>>> > Before the OutOfMemory this exception is found on all servers.
>>> >
>>> > Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall
>>> > search-central-prd-solr-temp1
>>> > ERROR: null:java.lang.NullPointerException
>>> search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
>>> > search-central-prd-solr-temp1
>>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
>>> > search-central-prd-solr-temp1
>>> > at
>>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>>> > search-central-prd-solr-temp1
>>> > at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> >
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>>> > search-central-prd-solr-temp1
>>> > at org.eclipse.jetty.server.Server.handle(Server.java:534)
>>> > search-central-prd-solr-temp1
>>> > at
>>> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>>> > search-central-prd-solr-temp1
>>> > at
>>> >
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>>> > search-central-prd-solr-temp1
>>> >
>>> > *Configuration*
>>> > Index size- 95 GB
>>> > Heap 30 GB
>>> > Ram-96 GB
>>> > Polling Interval - 3 Hours
>>> > Caching-
>>> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
>>> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache"
>>> size="30000"
>>> > initialSize="1000" autowarmCount="100" /> <documentCache class=
>>> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
>>> >
>>> > --
>>> >
>>> >
>>>
>>
>>
>>
--
Re: OutofMemory Error in solr 6.5
Posted by Dominique Bejean <do...@eolya.fr>.
Pretty sure the issue is caused by caches size at new searcher warmup time.
Dominique
Le mar. 10 août 2021 à 09:07, Satya Nand <sa...@indiamart.com> a
écrit :
> Hi Dominique,
>
> You don't provide information about the number of documents. Anyway, all
>> your cache size and mostly initial size are big. Cache are stored in JVM
>> heap.
>
> Document count is 101893353.
>
> About cache size, most is not always better. Did you make some performance
>> benchmarks in order to set these values ?
>
> We increased cache size in the hope to reduce some response time, We
> heavily use group queries with 7-8 boost factors. The average response time
> on this document set is 136 ms. We receive approx 65 requests/second in
> peak hours. The replication interval is 3 hours.
>
> The most strange thing about is that system keeps running for days without
> any issue, So I believe cache size should not be an issue. If the cache
> size had been the culprit, the issue would have been frequent. isn't it?
>
>
>
> On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean <do...@eolya.fr>
> wrote:
>
>> Hi,
>>
>> You don't provide information about the number of documents. Anyway, all
>> your cache size and mostly initial size are big. Cache are stored in JVM
>> heap.
>>
>> About cache size, most is not always better. Did you make some performance
>> benchmarks in order to set these values ?
>>
>> Try with the default values, after a few hours check cumulative caches
>> statistics in order to decide if you need to increase their sizes. The
>> objective is not to have cumulative_hitratio to 100%. There isn't ideal
>> value as it is really related to your datas, to the user's queries, to how
>> you build your queries ... but 70% is a good value. At some point
>> increasing the size again and again won't increase cumulative_hitratio a
>> lot as it is a logarithmic curve.
>>
>> Check also the heap usage also with your JVM GC logs and a tool like
>> gceasy.io
>>
>> Regards
>>
>> Dominique
>>
>>
>>
>>
>> Le lun. 9 août 2021 à 07:44, Satya Nand <satya.nand@indiamart.com
>> .invalid>
>> a écrit :
>>
>> > Hi,
>> > We are facing a strange issue in our solr system. Most of the days it
>> keeps
>> > running fine but once or twice in a month, we face OutofMemory on solr
>> > servers.
>> >
>> > We are using Leader-Follower architecture, one Leader and 4 followers.
>> > Strangely we get OutofMemory error on all follower servers.
>> > Before the OutOfMemory this exception is found on all servers.
>> >
>> > Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall
>> > search-central-prd-solr-temp1
>> > ERROR: null:java.lang.NullPointerException search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
>> > search-central-prd-solr-temp1
>> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
>> > search-central-prd-solr-temp1
>> > at
>> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>> > search-central-prd-solr-temp1
>> > at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> >
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>> > search-central-prd-solr-temp1
>> > at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> > search-central-prd-solr-temp1
>> > at
>> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>> > search-central-prd-solr-temp1
>> > at
>> >
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>> > search-central-prd-solr-temp1
>> >
>> > *Configuration*
>> > Index size- 95 GB
>> > Heap 30 GB
>> > Ram-96 GB
>> > Polling Interval - 3 Hours
>> > Caching-
>> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
>> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache"
>> size="30000"
>> > initialSize="1000" autowarmCount="100" /> <documentCache class=
>> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
>> >
>> > --
>> >
>> >
>>
>
>
>
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Dominique,
You don't provide information about the number of documents. Anyway, all
> your cache size and mostly initial size are big. Cache are stored in JVM
> heap.
Document count is 101893353.
About cache size, most is not always better. Did you make some performance
> benchmarks in order to set these values ?
We increased cache size in the hope to reduce some response time, We
heavily use group queries with 7-8 boost factors. The average response time
on this document set is 136 ms. We receive approx 65 requests/second in
peak hours. The replication interval is 3 hours.
The most strange thing about is that system keeps running for days without
any issue, So I believe cache size should not be an issue. If the cache
size had been the culprit, the issue would have been frequent. isn't it?
On Mon, Aug 9, 2021 at 6:44 PM Dominique Bejean <do...@eolya.fr>
wrote:
> Hi,
>
> You don't provide information about the number of documents. Anyway, all
> your cache size and mostly initial size are big. Cache are stored in JVM
> heap.
>
> About cache size, most is not always better. Did you make some performance
> benchmarks in order to set these values ?
>
> Try with the default values, after a few hours check cumulative caches
> statistics in order to decide if you need to increase their sizes. The
> objective is not to have cumulative_hitratio to 100%. There isn't ideal
> value as it is really related to your datas, to the user's queries, to how
> you build your queries ... but 70% is a good value. At some point
> increasing the size again and again won't increase cumulative_hitratio a
> lot as it is a logarithmic curve.
>
> Check also the heap usage also with your JVM GC logs and a tool like
> gceasy.io
>
> Regards
>
> Dominique
>
>
>
>
> Le lun. 9 août 2021 à 07:44, Satya Nand <sa...@indiamart.com.invalid>
> a écrit :
>
> > Hi,
> > We are facing a strange issue in our solr system. Most of the days it
> keeps
> > running fine but once or twice in a month, we face OutofMemory on solr
> > servers.
> >
> > We are using Leader-Follower architecture, one Leader and 4 followers.
> > Strangely we get OutofMemory error on all follower servers.
> > Before the OutOfMemory this exception is found on all servers.
> >
> > Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall
> > search-central-prd-solr-temp1
> > ERROR: null:java.lang.NullPointerException search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
> > search-central-prd-solr-temp1
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > search-central-prd-solr-temp1
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
> > search-central-prd-solr-temp1
> > at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> > search-central-prd-solr-temp1
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> > search-central-prd-solr-temp1
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > search-central-prd-solr-temp1
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> > search-central-prd-solr-temp1
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> > search-central-prd-solr-temp1
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > search-central-prd-solr-temp1
> > at org.eclipse.jetty.server.Server.handle(Server.java:534)
> > search-central-prd-solr-temp1
> > at
> > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> > search-central-prd-solr-temp1
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> > search-central-prd-solr-temp1
> >
> > *Configuration*
> > Index size- 95 GB
> > Heap 30 GB
> > Ram-96 GB
> > Polling Interval - 3 Hours
> > Caching-
> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache"
> size="30000"
> > initialSize="1000" autowarmCount="100" /> <documentCache class=
> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
> >
> > --
> >
> >
>
--
Re: OutofMemory Error in solr 6.5
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
You don't provide information about the number of documents. Anyway, all
your cache size and mostly initial size are big. Cache are stored in JVM
heap.
About cache size, most is not always better. Did you make some performance
benchmarks in order to set these values ?
Try with the default values, after a few hours check cumulative caches
statistics in order to decide if you need to increase their sizes. The
objective is not to have cumulative_hitratio to 100%. There isn't ideal
value as it is really related to your datas, to the user's queries, to how
you build your queries ... but 70% is a good value. At some point
increasing the size again and again won't increase cumulative_hitratio a
lot as it is a logarithmic curve.
Check also the heap usage also with your JVM GC logs and a tool like
gceasy.io
Regards
Dominique
Le lun. 9 août 2021 à 07:44, Satya Nand <sa...@indiamart.com.invalid>
a écrit :
> Hi,
> We are facing a strange issue in our solr system. Most of the days it keeps
> running fine but once or twice in a month, we face OutofMemory on solr
> servers.
>
> We are using Leader-Follower architecture, one Leader and 4 followers.
> Strangely we get OutofMemory error on all follower servers.
> Before the OutOfMemory this exception is found on all servers.
>
> Aug, 04 2021 15:26:11 org.apache.solr.servlet.HttpSolrCall
> search-central-prd-solr-temp1
> ERROR: null:java.lang.NullPointerException search-central-prd-solr-temp1
> at
>
> org.apache.solr.search.CollapsingQParserPlugin$OrdScoreCollector.finish(CollapsingQParserPlugin.java:617)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:240)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:2027)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
> search-central-prd-solr-temp1
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> search-central-prd-solr-temp1
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
> search-central-prd-solr-temp1
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> search-central-prd-solr-temp1
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> search-central-prd-solr-temp1
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> search-central-prd-solr-temp1
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> search-central-prd-solr-temp1
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> search-central-prd-solr-temp1
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> search-central-prd-solr-temp1
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> search-central-prd-solr-temp1
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> search-central-prd-solr-temp1
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> search-central-prd-solr-temp1
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> search-central-prd-solr-temp1
>
> *Configuration*
> Index size- 95 GB
> Heap 30 GB
> Ram-96 GB
> Polling Interval - 3 Hours
> Caching-
> <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
> autowarmCount="100" /> <queryResultCache class="solr.LRUCache" size="30000"
> initialSize="1000" autowarmCount="100" /> <documentCache class=
> "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
>
> --
>
>
Re: OutofMemory Error in solr 6.5
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Shawn,
>
>
> Do you have the actual OutOfMemoryError exception? Can we see that?
> There are several resources other than heap memory that will result in
> OOME if they are exhausted. It's important to be investigating the
> correct resource.
*Exception:*
Aug, 04 2021 15:38:36 org.apache.solr.servlet.HttpSolrCall
search-central-prd-solr-temp1
ERROR: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java
heap space search-central-prd-solr-temp1
at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:676)
search-central-prd-solr-temp1
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:544)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
search-central-prd-solr-temp1
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at org.eclipse.jetty.server.Server.handle(Server.java:534)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
search-central-prd-solr-temp1
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
search-central-prd-solr-temp1
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
search-central-prd-solr-temp1
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
search-central-prd-solr-temp1
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
search-central-prd-solr-temp1
at java.lang.Thread.run(Thread.java:748)
search-central-prd-solr-temp1
Caused by: java.lang.OutOfMemoryError: Java heap space
search-central-prd-solr-temp1
at org.apache.lucene.util.FixedBitSet.clone(FixedBitSet.java:480)
search-central-prd-solr-temp1
at
org.apache.solr.search.DocSetBase.intersection(DocSetBase.java:124)
search-central-prd-solr-temp1
at org.apache.solr.search.BitDocSet.intersection(BitDocSet.java:40)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1482)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1968)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1844)
search-central-prd-solr-temp1
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:609)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:547)
search-central-prd-solr-temp1
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
search-central-prd-solr-temp1
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
search-central-prd-solr-temp1
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
search-central-prd-solr-temp1
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
search-central-prd-solr-temp1
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
search-central-prd-solr-temp1
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
search-central-prd-solr-temp1
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
search-central-prd-solr-temp1
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
search-central-prd-solr-temp1
at org.eclipse.jetty.server.Server.handle(Server.java:534)
search-central-prd-solr-temp1
On Tue, Aug 10, 2021 at 6:30 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/8/2021 11:43 PM, Satya Nand wrote:
> > We are facing a strange issue in our solr system. Most of the days it
> keeps
> > running fine but once or twice in a month, we face OutofMemory on solr
> > servers.
> >
> > We are using Leader-Follower architecture, one Leader and 4 followers.
> > Strangely we get OutofMemory error on all follower servers.
> > Before the OutOfMemory this exception is found on all servers.
>
> Do you have the actual OutOfMemoryError exception? Can we see that?
> There are several resources other than heap memory that will result in
> OOME if they are exhausted. It's important to be investigating the
> correct resource.
>
> > <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
> > autowarmCount="100" /> <queryResultCache class="solr.LRUCache"
> size="30000"
> > initialSize="1000" autowarmCount="100" /> <documentCache class=
> > "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
>
> If you have five million documents (including those documents that have
> been deleted) in a core, then each filterCache entry for that core will
> be 625000 bytes, plus some unknown amount of overhead to manage the
> entry. Four thousand of them will consume 2.5 billion bytes. If you
> have multiple cores each with many documents, the amount of memory
> required for the filterCache could get VERY big.
>
> Until we can see the actual OOME exception, we won't know what resource
> you need to investigate. It is frequently NOT memory.
>
> Thanks,
> Shawn
>
--
Re: OutofMemory Error in solr 6.5
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/8/2021 11:43 PM, Satya Nand wrote:
> We are facing a strange issue in our solr system. Most of the days it keeps
> running fine but once or twice in a month, we face OutofMemory on solr
> servers.
>
> We are using Leader-Follower architecture, one Leader and 4 followers.
> Strangely we get OutofMemory error on all follower servers.
> Before the OutOfMemory this exception is found on all servers.
Do you have the actual OutOfMemoryError exception? Can we see that?
There are several resources other than heap memory that will result in
OOME if they are exhausted. It's important to be investigating the
correct resource.
> <filterCache class="solr.FastLRUCache" size="4000" initialSize="2000"
> autowarmCount="100" /> <queryResultCache class="solr.LRUCache" size="30000"
> initialSize="1000" autowarmCount="100" /> <documentCache class=
> "solr.LRUCache" size="25000" initialSize="512" autowarmCount="512" />
If you have five million documents (including those documents that have
been deleted) in a core, then each filterCache entry for that core will
be 625000 bytes, plus some unknown amount of overhead to manage the
entry. Four thousand of them will consume 2.5 billion bytes. If you
have multiple cores each with many documents, the amount of memory
required for the filterCache could get VERY big.
Until we can see the actual OOME exception, we won't know what resource
you need to investigate. It is frequently NOT memory.
Thanks,
Shawn