You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Satya Nand <sa...@indiamart.com.INVALID> on 2022/10/07 12:23:49 UTC
maximum searchers registered per replica in solr cloud.
Hi,
We have an 8-node cluster, Where each node contains only 1 replica of PULL
time.
For the last few hours suddenly we are facing high CPU usage and load on
one of the servers (double of the other servers)
Upon checking in the Solr Graphna Dashboard it was found that *Mapped Total
Capacity(Jvm Metrices->Buffer size section) *for this particular node was
approx double of other servers 54GB vs 28 GB.
Further checking in *CORE (Plugin/Stats) *for this particular server, There
were two searchers registered for this core. something like this
*Searcher@56aea1d7[im-search-03-08-22_shard2_replica_p19]
mainsearchercoreSearcher@6f3bd5b7[im-search-03-08-22_shard2_replica_p19]
main*
I checked other servers as well for the count of searchers, I only found
one for every core. for ex.
*searcherSearcher@7a7e7ba3[im-search-03-08-22_shard1_replica_p17] maincore*
Could these two searchers on one core have resulted in an increase in load
and CPU usage?
And what could have been the reason for two searchers, My understanding was
that only one searcher is there for 1 core.
If this is the issue can we do some configuration so that it doesn't
happen again?
Re: maximum searchers registered per replica in solr cloud.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/7/22 20:15, Satya Nand wrote:
> After sometime mapped total capacity reached to 100 GB(probably after some
> commit). Young GC count doubled and server threw out of memory error.
>
> Any insights? What we can check to avoid this?
Did you actually see the OutOfMemoryError exception? The exception will
include a note to say what resource was depleted. It is frequently NOT
memory. We need to chase the right resource.
I have been working on something in SOLR-8803 that will ensure we always
can see the cause of an OOME. The patch cannot be applied to Solr 8.x,
it has to go into 9.x. That's because the new option was added to Java
8u92 and with Solr 8.x we cannot guarantee that the user has a new
enough Java version. For Solr 9.x we can, because it requires Java 11.
https://issues.apache.org/jira/browse/SOLR-8803
The parts of the patch that affect the bin directory can be applied to a
binary download to get the new OOME behavior.
Thanks,
Shawn
Re: maximum searchers registered per replica in solr cloud.
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Shawn,
After sometime mapped total capacity reached to 100 GB(probably after some
commit). Young GC count doubled and server threw out of memory error.
Any insights? What we can check to avoid this?
On Fri, 7 Oct, 2022, 8:03 pm Satya Nand, <sa...@indiamart.com> wrote:
> Shawn,
>
> Yes earlier I did not find out about the multiple searchers. I reached to
> this after debugging a lot.
> And now i can confirm it was the same searcher when i posted the question
> first time.
>
> I reloaded the collection 15 minutes ago and now the stuck searcher is
> gone. So i won’t be able to tell you the warmup time for stuck searcher.
>
> But after reloading now the mapped total capacity has reached to 75 GB. On
> other servers it is still 25 GB.
>
>
> We do use our custom phonetic implementation. But it is running since
> years.
>
>
>
>
> On Fri, 7 Oct 2022 at 7:39 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 10/7/22 07:55, Satya Nand wrote:
>> > I guess it is something related to mmap directory implementation of
>> index
>> > directory though I could be wrong.
>>
>> That makes sense. MMAP space is not a count of memory actually taken,
>> just the amount of data that is accessible. The OS manages how much of
>> that data is actually in memory.
>>
>> > These 2 searchers are there since 11 am. And after every commit two more
>> > searchers are being opened again.
>>
>> I don't know what timezone you are in, but I am assuming that was quite
>> a while ago. Are you sure it's the exact same searcher that you saw at
>> 11 AM? What is the warmupTime for each searcher you see in Plugins/Stats?
>>
>> Just saw your additional message. Are you running with any modules or
>> custom code? Something that required loading extra jars?
>>
>> Thanks,
>> Shawn
>>
>>
Re: maximum searchers registered per replica in solr cloud.
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Shawn,
Yes earlier I did not find out about the multiple searchers. I reached to
this after debugging a lot.
And now i can confirm it was the same searcher when i posted the question
first time.
I reloaded the collection 15 minutes ago and now the stuck searcher is
gone. So i won’t be able to tell you the warmup time for stuck searcher.
But after reloading now the mapped total capacity has reached to 75 GB. On
other servers it is still 25 GB.
We do use our custom phonetic implementation. But it is running since years.
On Fri, 7 Oct 2022 at 7:39 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 10/7/22 07:55, Satya Nand wrote:
> > I guess it is something related to mmap directory implementation of index
> > directory though I could be wrong.
>
> That makes sense. MMAP space is not a count of memory actually taken,
> just the amount of data that is accessible. The OS manages how much of
> that data is actually in memory.
>
> > These 2 searchers are there since 11 am. And after every commit two more
> > searchers are being opened again.
>
> I don't know what timezone you are in, but I am assuming that was quite
> a while ago. Are you sure it's the exact same searcher that you saw at
> 11 AM? What is the warmupTime for each searcher you see in Plugins/Stats?
>
> Just saw your additional message. Are you running with any modules or
> custom code? Something that required loading extra jars?
>
> Thanks,
> Shawn
>
>
Re: maximum searchers registered per replica in solr cloud.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/7/22 07:55, Satya Nand wrote:
> I guess it is something related to mmap directory implementation of index
> directory though I could be wrong.
That makes sense. MMAP space is not a count of memory actually taken,
just the amount of data that is accessible. The OS manages how much of
that data is actually in memory.
> These 2 searchers are there since 11 am. And after every commit two more
> searchers are being opened again.
I don't know what timezone you are in, but I am assuming that was quite
a while ago. Are you sure it's the exact same searcher that you saw at
11 AM? What is the warmupTime for each searcher you see in Plugins/Stats?
Just saw your additional message. Are you running with any modules or
custom code? Something that required loading extra jars?
Thanks,
Shawn
Re: maximum searchers registered per replica in solr cloud.
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Shawn,
I just observed that 2nd searcher is not getting closed. It was also there
when I originally posted the question.
On Fri, 7 Oct 2022 at 7:25 PM, Satya Nand <sa...@indiamart.com> wrote:
> Hi Shawn,
>
> I dont think it was heap size. We have assigned only 8 GB heap size. This
> was named as solr mapped total capacity in grafana dashboard . Heap size
> section was also there where i could see the heap usages based on 8 gb. But
> gc count and time for this server was high as well.
>
> I guess it is something related to mmap directory implementation of index
> directory though I could be wrong.
>
> These 2 searchers are there since 11 am. And after every commit two more
> searchers are being opened again.
>
> Right now these are two searchers.
>
> -
> Searcher@479c8248[im-search-03-08-22_shard2_replica_p19] main
> - Searcher@6f3bd5b7[im-search-03-08-22_shard2_replica_p19] main
>
> Sharing the cache configuration as well. Total documents in this
> replica are 18 million.(25 mn maxdoc, 7 mn deleted doc.
> <filterCache class="solr.CaffeineCache" size="1000" initialSize="300"
> autowarmCount="100" /> <queryResultCache class="solr.CaffeineCache"
> size="30000" initialSize="1000" autowarmCount="100" /> <documentCache
> class="solr.CaffeineCache" size="25000" initialSize="512" autowarmCount
> ="512" />
>
>
>
>
>
> On Fri, 7 Oct 2022 at 6:32 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 10/7/22 06:23, Satya Nand wrote:
>> > Upon checking in the Solr Graphna Dashboard it was found that *Mapped
>> Total
>> > Capacity(Jvm Metrices->Buffer size section) *for this particular node
>> was
>> > approx double of other servers 54GB vs 28 GB.
>> >
>> > Further checking in *CORE (Plugin/Stats) *for this particular server,
>> There
>> > were two searchers registered for this core. something like this
>>
>> Usually when there are multiple searchers, it's because there is an
>> existing searcher handling queries and at least one new searcher that is
>> being warmed as a replacement. When the new searcher is fully warmed,
>> the existing searcher will shut down as soon as all queries that are
>> using it are complete.
>>
>> 28GB of heap memory being assigned to the searcher seems extremely
>> excessive. Can you share the cache configuration in solrconfig.xml and
>> the max doc count in the core?
>>
>> Thanks,
>> Shawn
>>
>>
Re: maximum searchers registered per replica in solr cloud.
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Shawn,
I dont think it was heap size. We have assigned only 8 GB heap size. This
was named as solr mapped total capacity in grafana dashboard . Heap size
section was also there where i could see the heap usages based on 8 gb. But
gc count and time for this server was high as well.
I guess it is something related to mmap directory implementation of index
directory though I could be wrong.
These 2 searchers are there since 11 am. And after every commit two more
searchers are being opened again.
Right now these are two searchers.
-
Searcher@479c8248[im-search-03-08-22_shard2_replica_p19] main
- Searcher@6f3bd5b7[im-search-03-08-22_shard2_replica_p19] main
Sharing the cache configuration as well. Total documents in this replica
are 18 million.(25 mn maxdoc, 7 mn deleted doc.
<filterCache class="solr.CaffeineCache" size="1000" initialSize="300"
autowarmCount="100" /> <queryResultCache class="solr.CaffeineCache" size=
"30000" initialSize="1000" autowarmCount="100" /> <documentCache class=
"solr.CaffeineCache" size="25000" initialSize="512" autowarmCount="512"
/>
On Fri, 7 Oct 2022 at 6:32 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 10/7/22 06:23, Satya Nand wrote:
> > Upon checking in the Solr Graphna Dashboard it was found that *Mapped
> Total
> > Capacity(Jvm Metrices->Buffer size section) *for this particular node was
> > approx double of other servers 54GB vs 28 GB.
> >
> > Further checking in *CORE (Plugin/Stats) *for this particular server,
> There
> > were two searchers registered for this core. something like this
>
> Usually when there are multiple searchers, it's because there is an
> existing searcher handling queries and at least one new searcher that is
> being warmed as a replacement. When the new searcher is fully warmed,
> the existing searcher will shut down as soon as all queries that are
> using it are complete.
>
> 28GB of heap memory being assigned to the searcher seems extremely
> excessive. Can you share the cache configuration in solrconfig.xml and
> the max doc count in the core?
>
> Thanks,
> Shawn
>
>
Re: maximum searchers registered per replica in solr cloud.
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/7/22 06:23, Satya Nand wrote:
> Upon checking in the Solr Graphna Dashboard it was found that *Mapped Total
> Capacity(Jvm Metrices->Buffer size section) *for this particular node was
> approx double of other servers 54GB vs 28 GB.
>
> Further checking in *CORE (Plugin/Stats) *for this particular server, There
> were two searchers registered for this core. something like this
Usually when there are multiple searchers, it's because there is an
existing searcher handling queries and at least one new searcher that is
being warmed as a replacement. When the new searcher is fully warmed,
the existing searcher will shut down as soon as all queries that are
using it are complete.
28GB of heap memory being assigned to the searcher seems extremely
excessive. Can you share the cache configuration in solrconfig.xml and
the max doc count in the core?
Thanks,
Shawn