You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Jeff <mc...@gmail.com> on 2021/08/08 19:43:19 UTC
Cpu 100%
Hello,
I am currently running a 20 node Solrcloud on 7.7
We have 800,000,000 documents
I am experiencing CPU spikes well over 100%
Adding more CPU did not solve the issue.
searcherExecutor times are ballooning.
I have 2 replicas per shard. 1 NRT and 1 TLOG.
We are indexing around 200,000 per hour.
I have disabled soft commit.
Any help is greatly appreciated!
<maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
<openSearcher>true</openSearcher>
Re: Cpu 100%
Posted by sudhir kumar <go...@gmail.com>.
Hi Jeff,
Few things to look over
1) what’s the document size and how many fields are indexed and what is the
type of fields indexed.
2) are you sending explicit commits while updating the documents.
3) make opensearcher false and try to index the data.
Thanks
On Mon, Aug 9, 2021 at 10:47 AM Deepak Goel <de...@gmail.com> wrote:
> Looks like your SolrCloud is running in a single-threaded mode. So adding
> CPU's might not help!
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
>
> > Hello,
> > I am currently running a 20 node Solrcloud on 7.7
> >
> > We have 800,000,000 documents
> >
> > I am experiencing CPU spikes well over 100%
> > Adding more CPU did not solve the issue.
> >
> > searcherExecutor times are ballooning.
> >
> > I have 2 replicas per shard. 1 NRT and 1 TLOG.
> >
> > We are indexing around 200,000 per hour.
> >
> > I have disabled soft commit.
> >
> > Any help is greatly appreciated!
> >
> > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> > <openSearcher>true</openSearcher>
> >
>
Re: Cpu 100%
Posted by Rahul Goswami <ra...@gmail.com>.
Jeff,
Can you please share the output of this REST end point from the node where
you are experiencing high CPU usage? Do this while you are seeing the high
CPU usage. Preferably share 3 samples at intervals of 10 seconds .
localhost:8983/solr/admin/info/threads
Also, during which operation are you seeing the high CPU usage?
-Rahul
On Mon, Aug 9, 2021 at 6:00 PM Dominique Bejean <do...@eolya.fr>
wrote:
> Ok, I got it. You mean job that send data to Solr for indexing is single
> thread, not Solr itself :)
>
> Dominique
>
> Le lun. 9 août 2021 à 17:19, Deepak Goel <de...@gmail.com> a écrit :
>
> >
> >
> https://hybrisarchitect.com/solr-configuration-increase-number-of-indexer-threads/
> >
> > On Mon, 9 Aug 2021, 18:04 Dominique Bejean, <do...@eolya.fr>
> > wrote:
> >
> >> Hi Deepak,
> >>
> >> Can you explain Solr single-threaded mode ?
> >> I am not aware of any settings about a single-threaded mode !
> >>
> >> Regards
> >>
> >> Dominique
> >>
> >>
> >> Le lun. 9 août 2021 à 07:16, Deepak Goel <de...@gmail.com> a écrit :
> >>
> >> > Looks like your SolrCloud is running in a single-threaded mode. So
> >> adding
> >> > CPU's might not help!
> >> >
> >> >
> >> > Deepak
> >> > "The greatness of a nation can be judged by the way its animals are
> >> treated
> >> > - Mahatma Gandhi"
> >> >
> >> > +91 73500 12833
> >> > deicool@gmail.com
> >> >
> >> > Facebook: https://www.facebook.com/deicool
> >> > LinkedIn: www.linkedin.com/in/deicool
> >> >
> >> > "Plant a Tree, Go Green"
> >> >
> >> > Make In India : http://www.makeinindia.com/home
> >> >
> >> >
> >> > On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
> >> >
> >> > > Hello,
> >> > > I am currently running a 20 node Solrcloud on 7.7
> >> > >
> >> > > We have 800,000,000 documents
> >> > >
> >> > > I am experiencing CPU spikes well over 100%
> >> > > Adding more CPU did not solve the issue.
> >> > >
> >> > > searcherExecutor times are ballooning.
> >> > >
> >> > > I have 2 replicas per shard. 1 NRT and 1 TLOG.
> >> > >
> >> > > We are indexing around 200,000 per hour.
> >> > >
> >> > > I have disabled soft commit.
> >> > >
> >> > > Any help is greatly appreciated!
> >> > >
> >> > > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> >> > > <openSearcher>true</openSearcher>
> >> > >
> >> >
> >>
> >
>
Re: Cpu 100%
Posted by Dominique Bejean <do...@eolya.fr>.
Ok, I got it. You mean job that send data to Solr for indexing is single
thread, not Solr itself :)
Dominique
Le lun. 9 août 2021 à 17:19, Deepak Goel <de...@gmail.com> a écrit :
>
> https://hybrisarchitect.com/solr-configuration-increase-number-of-indexer-threads/
>
> On Mon, 9 Aug 2021, 18:04 Dominique Bejean, <do...@eolya.fr>
> wrote:
>
>> Hi Deepak,
>>
>> Can you explain Solr single-threaded mode ?
>> I am not aware of any settings about a single-threaded mode !
>>
>> Regards
>>
>> Dominique
>>
>>
>> Le lun. 9 août 2021 à 07:16, Deepak Goel <de...@gmail.com> a écrit :
>>
>> > Looks like your SolrCloud is running in a single-threaded mode. So
>> adding
>> > CPU's might not help!
>> >
>> >
>> > Deepak
>> > "The greatness of a nation can be judged by the way its animals are
>> treated
>> > - Mahatma Gandhi"
>> >
>> > +91 73500 12833
>> > deicool@gmail.com
>> >
>> > Facebook: https://www.facebook.com/deicool
>> > LinkedIn: www.linkedin.com/in/deicool
>> >
>> > "Plant a Tree, Go Green"
>> >
>> > Make In India : http://www.makeinindia.com/home
>> >
>> >
>> > On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
>> >
>> > > Hello,
>> > > I am currently running a 20 node Solrcloud on 7.7
>> > >
>> > > We have 800,000,000 documents
>> > >
>> > > I am experiencing CPU spikes well over 100%
>> > > Adding more CPU did not solve the issue.
>> > >
>> > > searcherExecutor times are ballooning.
>> > >
>> > > I have 2 replicas per shard. 1 NRT and 1 TLOG.
>> > >
>> > > We are indexing around 200,000 per hour.
>> > >
>> > > I have disabled soft commit.
>> > >
>> > > Any help is greatly appreciated!
>> > >
>> > > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
>> > > <openSearcher>true</openSearcher>
>> > >
>> >
>>
>
Re: Cpu 100%
Posted by Deepak Goel <de...@gmail.com>.
https://hybrisarchitect.com/solr-configuration-increase-number-of-indexer-threads/
On Mon, 9 Aug 2021, 18:04 Dominique Bejean, <do...@eolya.fr>
wrote:
> Hi Deepak,
>
> Can you explain Solr single-threaded mode ?
> I am not aware of any settings about a single-threaded mode !
>
> Regards
>
> Dominique
>
>
> Le lun. 9 août 2021 à 07:16, Deepak Goel <de...@gmail.com> a écrit :
>
> > Looks like your SolrCloud is running in a single-threaded mode. So adding
> > CPU's might not help!
> >
> >
> > Deepak
> > "The greatness of a nation can be judged by the way its animals are
> treated
> > - Mahatma Gandhi"
> >
> > +91 73500 12833
> > deicool@gmail.com
> >
> > Facebook: https://www.facebook.com/deicool
> > LinkedIn: www.linkedin.com/in/deicool
> >
> > "Plant a Tree, Go Green"
> >
> > Make In India : http://www.makeinindia.com/home
> >
> >
> > On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
> >
> > > Hello,
> > > I am currently running a 20 node Solrcloud on 7.7
> > >
> > > We have 800,000,000 documents
> > >
> > > I am experiencing CPU spikes well over 100%
> > > Adding more CPU did not solve the issue.
> > >
> > > searcherExecutor times are ballooning.
> > >
> > > I have 2 replicas per shard. 1 NRT and 1 TLOG.
> > >
> > > We are indexing around 200,000 per hour.
> > >
> > > I have disabled soft commit.
> > >
> > > Any help is greatly appreciated!
> > >
> > > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> > > <openSearcher>true</openSearcher>
> > >
> >
>
Re: Cpu 100%
Posted by Dominique Bejean <do...@eolya.fr>.
Hi Deepak,
Can you explain Solr single-threaded mode ?
I am not aware of any settings about a single-threaded mode !
Regards
Dominique
Le lun. 9 août 2021 à 07:16, Deepak Goel <de...@gmail.com> a écrit :
> Looks like your SolrCloud is running in a single-threaded mode. So adding
> CPU's might not help!
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
>
> > Hello,
> > I am currently running a 20 node Solrcloud on 7.7
> >
> > We have 800,000,000 documents
> >
> > I am experiencing CPU spikes well over 100%
> > Adding more CPU did not solve the issue.
> >
> > searcherExecutor times are ballooning.
> >
> > I have 2 replicas per shard. 1 NRT and 1 TLOG.
> >
> > We are indexing around 200,000 per hour.
> >
> > I have disabled soft commit.
> >
> > Any help is greatly appreciated!
> >
> > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> > <openSearcher>true</openSearcher>
> >
>
Re: Cpu 100%
Posted by Deepak Goel <de...@gmail.com>.
Looks like your SolrCloud is running in a single-threaded mode. So adding
CPU's might not help!
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Mon, Aug 9, 2021 at 1:13 AM Jeff <mc...@gmail.com> wrote:
> Hello,
> I am currently running a 20 node Solrcloud on 7.7
>
> We have 800,000,000 documents
>
> I am experiencing CPU spikes well over 100%
> Adding more CPU did not solve the issue.
>
> searcherExecutor times are ballooning.
>
> I have 2 replicas per shard. 1 NRT and 1 TLOG.
>
> We are indexing around 200,000 per hour.
>
> I have disabled soft commit.
>
> Any help is greatly appreciated!
>
> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> <openSearcher>true</openSearcher>
>
Re: Cpu 100%
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/8/2021 1:43 PM, Jeff wrote:
> I am currently running a 20 node Solrcloud on 7.7
>
> We have 800,000,000 documents
>
> I am experiencing CPU spikes well over 100%
> Adding more CPU did not solve the issue.
What OS is Solr running on? What tool are you looking at that says 100
percent CPU? For some operating systems and tools, 100 percent might be
very low.
Why have you flipped openSearcher to true on autoCommit? A 60 second
commit maxTime when you are opening a new searcher is fairly aggressive.
What I would probably do here is a 60 second autoCommit with
openSearcher set to false, and autoSoftCommit on a longer interval,
something like five minutes.
How long are your commits taking to warm up completely? To see how long
the last commit took in the admin UI, choose the core from the dropdown,
then navigate through these options: Plugins/Stats, CORE, searcher.
Then look for something in the list that includes warmupTime. It will
probably say "SEARCHER.searcher.warmupTime" and be near the end of the
list. That number should be in milliseconds.
Thanks,
Shawn
Re: Cpu 100%
Posted by Michael Gibney <mi...@michaelgibney.net>.
Jeff, you mention indexing, but I'm curious: is this also a live system
supporting queries at the same time?
On Mon, Aug 9, 2021 at 8:53 AM Dominique Bejean <do...@eolya.fr>
wrote:
> Hi Jeff,
>
> How many CPU ?
> What is the CPU load average (information provided by Linux top command in
> the first line) ?
>
> I don't think it is the root cause of your issue, but 1 NRT + 1 TLOG
> replica per shard is not a usual practice. You have to create 2 TLOG
> replicas per shard and let Solr elect the NRT one.
>
> Regards
>
> Dominique
>
>
>
>
> Le dim. 8 août 2021 à 21:42, Jeff <mc...@gmail.com> a écrit :
>
> > Hello,
> > I am currently running a 20 node Solrcloud on 7.7
> >
> > We have 800,000,000 documents
> >
> > I am experiencing CPU spikes well over 100%
> > Adding more CPU did not solve the issue.
> >
> > searcherExecutor times are ballooning.
> >
> > I have 2 replicas per shard. 1 NRT and 1 TLOG.
> >
> > We are indexing around 200,000 per hour.
> >
> > I have disabled soft commit.
> >
> > Any help is greatly appreciated!
> >
> > <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> > <openSearcher>true</openSearcher>
> >
>
Re: Cpu 100%
Posted by Dominique Bejean <do...@eolya.fr>.
Hi Jeff,
How many CPU ?
What is the CPU load average (information provided by Linux top command in
the first line) ?
I don't think it is the root cause of your issue, but 1 NRT + 1 TLOG
replica per shard is not a usual practice. You have to create 2 TLOG
replicas per shard and let Solr elect the NRT one.
Regards
Dominique
Le dim. 8 août 2021 à 21:42, Jeff <mc...@gmail.com> a écrit :
> Hello,
> I am currently running a 20 node Solrcloud on 7.7
>
> We have 800,000,000 documents
>
> I am experiencing CPU spikes well over 100%
> Adding more CPU did not solve the issue.
>
> searcherExecutor times are ballooning.
>
> I have 2 replicas per shard. 1 NRT and 1 TLOG.
>
> We are indexing around 200,000 per hour.
>
> I have disabled soft commit.
>
> Any help is greatly appreciated!
>
> <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>
> <openSearcher>true</openSearcher>
>