You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by HariBabu kuruva <ha...@gmail.com> on 2023/03/24 10:23:13 UTC
Solr Node High CPU
Hi All,
We have 4 solr nodes, only one of the nodes is consuming almost High CPU
from yesterday. We don't see any errors in the logs. Restart dint resolve
the issue. Could you please help me to identify the issue.
Solr Version: 8.11.1
Instances: Single instance per machine. No other softwares running on the
machine.
CPU: 6 CPU's
Heap: 30GB per Instance
RAM: 60 GB
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr Node High CPU
Posted by Joel Bernstein <jo...@gmail.com>.
It was mentioned earlier but the thread dump is a good place to start
looking. Typically you'd see a number of threads doing something CPU
intensive. After that the heap dump will show you what parameters were used
in the threads that running.
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Mar 27, 2023 at 5:26 AM Deepak Goel <de...@gmail.com> wrote:
> can you run the following on the machines:
>
> dpkg -s
>
> (see if the versions of all libraries match)
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Sun, Mar 26, 2023 at 5:05 PM HariBabu kuruva <hari2708.kuruva@gmail.com
> >
> wrote:
>
> > Hi All,
> >
> > The only error i can see in the logs is below :
> >
> >
> >
> > *2023-03-26 11:28:11.352 ERROR (qtp1302725372-100592) [c:dqmStore
> s:shard1
> > r:core_node16 x:dqmStore_shard1_replica_n15] o.a.s.h.RequestHandlerBase
> > org.apache.solr.common.SolrException: undefined field:
> "openApplicationInc"
> > => org.apache.solr.common.SolrException: undefined field:
> > "openApplicationInc"*
> >
> > Is there any chance this error might be causing the issue ? But I can see
> > this error in the Leader node of the dqmStore where there is no CPU
> issue.
> >
> > This is not the new installation, it has been running for quite a year.
> >
> > We are using NRT replicas.
> >
> >
> >
> >
> >
> >
> > On Sun, Mar 26, 2023 at 3:07 PM Vivaldi <uy...@vivaldi.net.invalid>
> > wrote:
> >
> > > Are you using tlog replicas? Is that node the leader doing a lot of
> > > indexing?
> > >
> > > > On 25 Mar 2023, at 16:52, Deepak Goel <de...@gmail.com> wrote:
> > > >
> > > > It is quite possible, the installation on the server (high cpu) has
> > gone
> > > > astray. You might want to reinstall it (including updating all
> > > libraries).
> > > >
> > > >> On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, <mk...@apache.org>
> wrote:
> > > >>
> > > >> SolrAdmin Thread Dump, or jstack might give a clue.
> > > >>
> > > >> On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <
> > > hari2708.kuruva@gmail.com
> > > >>>
> > > >> wrote:
> > > >>
> > > >>> It's the solr process consuming the 95% CPU. I want to know how to
> > > >>> troubleshoot this from the solr end.
> > > >>>
> > > >>>> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> can you check which process is eating up the CPU using top command
> > > >>>>
> > > >>>> On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <
> > > hari2708.kuruva@gmail.com
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi All,
> > > >>>>>
> > > >>>>> We have 4 solr nodes, only one of the nodes is consuming almost
> > High
> > > >>> CPU
> > > >>>>> from yesterday. We don't see any errors in the logs. Restart dint
> > > >>> resolve
> > > >>>>> the issue. Could you please help me to identify the issue.
> > > >>>>>
> > > >>>>> Solr Version: 8.11.1
> > > >>>>> Instances: Single instance per machine. No other softwares
> running
> > on
> > > >>> the
> > > >>>>> machine.
> > > >>>>> CPU: 6 CPU's
> > > >>>>> Heap: 30GB per Instance
> > > >>>>> RAM: 60 GB
> > > >>>>>
> > > >>>>> --
> > > >>>>>
> > > >>>>> Thanks and Regards,
> > > >>>>> Hari
> > > >>>>> Mobile:9790756568
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>>
> > > >>> Thanks and Regards,
> > > >>> Hari
> > > >>> Mobile:9790756568
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >> Sincerely yours
> > > >> Mikhail Khludnev
> > > >> https://t.me/MUST_SEARCH
> > > >> A caveat: Cyrillic!
> > > >>
> > >
> > >
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
Re: Solr Node High CPU
Posted by Deepak Goel <de...@gmail.com>.
can you run the following on the machines:
dpkg -s
(see if the versions of all libraries match)
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Sun, Mar 26, 2023 at 5:05 PM HariBabu kuruva <ha...@gmail.com>
wrote:
> Hi All,
>
> The only error i can see in the logs is below :
>
>
>
> *2023-03-26 11:28:11.352 ERROR (qtp1302725372-100592) [c:dqmStore s:shard1
> r:core_node16 x:dqmStore_shard1_replica_n15] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: undefined field: "openApplicationInc"
> => org.apache.solr.common.SolrException: undefined field:
> "openApplicationInc"*
>
> Is there any chance this error might be causing the issue ? But I can see
> this error in the Leader node of the dqmStore where there is no CPU issue.
>
> This is not the new installation, it has been running for quite a year.
>
> We are using NRT replicas.
>
>
>
>
>
>
> On Sun, Mar 26, 2023 at 3:07 PM Vivaldi <uy...@vivaldi.net.invalid>
> wrote:
>
> > Are you using tlog replicas? Is that node the leader doing a lot of
> > indexing?
> >
> > > On 25 Mar 2023, at 16:52, Deepak Goel <de...@gmail.com> wrote:
> > >
> > > It is quite possible, the installation on the server (high cpu) has
> gone
> > > astray. You might want to reinstall it (including updating all
> > libraries).
> > >
> > >> On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, <mk...@apache.org> wrote:
> > >>
> > >> SolrAdmin Thread Dump, or jstack might give a clue.
> > >>
> > >> On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <
> > hari2708.kuruva@gmail.com
> > >>>
> > >> wrote:
> > >>
> > >>> It's the solr process consuming the 95% CPU. I want to know how to
> > >>> troubleshoot this from the solr end.
> > >>>
> > >>>> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com>
> > wrote:
> > >>>>
> > >>>> can you check which process is eating up the CPU using top command
> > >>>>
> > >>>> On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <
> > hari2708.kuruva@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi All,
> > >>>>>
> > >>>>> We have 4 solr nodes, only one of the nodes is consuming almost
> High
> > >>> CPU
> > >>>>> from yesterday. We don't see any errors in the logs. Restart dint
> > >>> resolve
> > >>>>> the issue. Could you please help me to identify the issue.
> > >>>>>
> > >>>>> Solr Version: 8.11.1
> > >>>>> Instances: Single instance per machine. No other softwares running
> on
> > >>> the
> > >>>>> machine.
> > >>>>> CPU: 6 CPU's
> > >>>>> Heap: 30GB per Instance
> > >>>>> RAM: 60 GB
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> Thanks and Regards,
> > >>>>> Hari
> > >>>>> Mobile:9790756568
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>> --
> > >>>
> > >>> Thanks and Regards,
> > >>> Hari
> > >>> Mobile:9790756568
> > >>>
> > >>
> > >>
> > >> --
> > >> Sincerely yours
> > >> Mikhail Khludnev
> > >> https://t.me/MUST_SEARCH
> > >> A caveat: Cyrillic!
> > >>
> >
> >
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
Re: Solr Node High CPU
Posted by HariBabu kuruva <ha...@gmail.com>.
Hi All,
The only error i can see in the logs is below :
*2023-03-26 11:28:11.352 ERROR (qtp1302725372-100592) [c:dqmStore s:shard1
r:core_node16 x:dqmStore_shard1_replica_n15] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: undefined field: "openApplicationInc"
=> org.apache.solr.common.SolrException: undefined field:
"openApplicationInc"*
Is there any chance this error might be causing the issue ? But I can see
this error in the Leader node of the dqmStore where there is no CPU issue.
This is not the new installation, it has been running for quite a year.
We are using NRT replicas.
On Sun, Mar 26, 2023 at 3:07 PM Vivaldi <uy...@vivaldi.net.invalid> wrote:
> Are you using tlog replicas? Is that node the leader doing a lot of
> indexing?
>
> > On 25 Mar 2023, at 16:52, Deepak Goel <de...@gmail.com> wrote:
> >
> > It is quite possible, the installation on the server (high cpu) has gone
> > astray. You might want to reinstall it (including updating all
> libraries).
> >
> >> On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, <mk...@apache.org> wrote:
> >>
> >> SolrAdmin Thread Dump, or jstack might give a clue.
> >>
> >> On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <
> hari2708.kuruva@gmail.com
> >>>
> >> wrote:
> >>
> >>> It's the solr process consuming the 95% CPU. I want to know how to
> >>> troubleshoot this from the solr end.
> >>>
> >>>> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com>
> wrote:
> >>>>
> >>>> can you check which process is eating up the CPU using top command
> >>>>
> >>>> On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <
> hari2708.kuruva@gmail.com
> >>>
> >>>> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> We have 4 solr nodes, only one of the nodes is consuming almost High
> >>> CPU
> >>>>> from yesterday. We don't see any errors in the logs. Restart dint
> >>> resolve
> >>>>> the issue. Could you please help me to identify the issue.
> >>>>>
> >>>>> Solr Version: 8.11.1
> >>>>> Instances: Single instance per machine. No other softwares running on
> >>> the
> >>>>> machine.
> >>>>> CPU: 6 CPU's
> >>>>> Heap: 30GB per Instance
> >>>>> RAM: 60 GB
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Thanks and Regards,
> >>>>> Hari
> >>>>> Mobile:9790756568
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> Thanks and Regards,
> >>> Hari
> >>> Mobile:9790756568
> >>>
> >>
> >>
> >> --
> >> Sincerely yours
> >> Mikhail Khludnev
> >> https://t.me/MUST_SEARCH
> >> A caveat: Cyrillic!
> >>
>
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr Node High CPU
Posted by Vivaldi <uy...@vivaldi.net.INVALID>.
Are you using tlog replicas? Is that node the leader doing a lot of indexing?
> On 25 Mar 2023, at 16:52, Deepak Goel <de...@gmail.com> wrote:
>
> It is quite possible, the installation on the server (high cpu) has gone
> astray. You might want to reinstall it (including updating all libraries).
>
>> On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, <mk...@apache.org> wrote:
>>
>> SolrAdmin Thread Dump, or jstack might give a clue.
>>
>> On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <hari2708.kuruva@gmail.com
>>>
>> wrote:
>>
>>> It's the solr process consuming the 95% CPU. I want to know how to
>>> troubleshoot this from the solr end.
>>>
>>>> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com> wrote:
>>>>
>>>> can you check which process is eating up the CPU using top command
>>>>
>>>> On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <hari2708.kuruva@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> We have 4 solr nodes, only one of the nodes is consuming almost High
>>> CPU
>>>>> from yesterday. We don't see any errors in the logs. Restart dint
>>> resolve
>>>>> the issue. Could you please help me to identify the issue.
>>>>>
>>>>> Solr Version: 8.11.1
>>>>> Instances: Single instance per machine. No other softwares running on
>>> the
>>>>> machine.
>>>>> CPU: 6 CPU's
>>>>> Heap: 30GB per Instance
>>>>> RAM: 60 GB
>>>>>
>>>>> --
>>>>>
>>>>> Thanks and Regards,
>>>>> Hari
>>>>> Mobile:9790756568
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Thanks and Regards,
>>> Hari
>>> Mobile:9790756568
>>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> https://t.me/MUST_SEARCH
>> A caveat: Cyrillic!
>>
Re: Solr Node High CPU
Posted by Deepak Goel <de...@gmail.com>.
It is quite possible, the installation on the server (high cpu) has gone
astray. You might want to reinstall it (including updating all libraries).
On Fri, 24 Mar 2023, 18:31 Mikhail Khludnev, <mk...@apache.org> wrote:
> SolrAdmin Thread Dump, or jstack might give a clue.
>
> On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <hari2708.kuruva@gmail.com
> >
> wrote:
>
> > It's the solr process consuming the 95% CPU. I want to know how to
> > troubleshoot this from the solr end.
> >
> > On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com> wrote:
> >
> > > can you check which process is eating up the CPU using top command
> > >
> > > On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <hari2708.kuruva@gmail.com
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We have 4 solr nodes, only one of the nodes is consuming almost High
> > CPU
> > > > from yesterday. We don't see any errors in the logs. Restart dint
> > resolve
> > > > the issue. Could you please help me to identify the issue.
> > > >
> > > > Solr Version: 8.11.1
> > > > Instances: Single instance per machine. No other softwares running on
> > the
> > > > machine.
> > > > CPU: 6 CPU's
> > > > Heap: 30GB per Instance
> > > > RAM: 60 GB
> > > >
> > > > --
> > > >
> > > > Thanks and Regards,
> > > > Hari
> > > > Mobile:9790756568
> > > >
> > >
> >
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> https://t.me/MUST_SEARCH
> A caveat: Cyrillic!
>
Re: Solr Node High CPU
Posted by Mikhail Khludnev <mk...@apache.org>.
SolrAdmin Thread Dump, or jstack might give a clue.
On Fri, Mar 24, 2023 at 3:41 PM HariBabu kuruva <ha...@gmail.com>
wrote:
> It's the solr process consuming the 95% CPU. I want to know how to
> troubleshoot this from the solr end.
>
> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com> wrote:
>
> > can you check which process is eating up the CPU using top command
> >
> > On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <ha...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > We have 4 solr nodes, only one of the nodes is consuming almost High
> CPU
> > > from yesterday. We don't see any errors in the logs. Restart dint
> resolve
> > > the issue. Could you please help me to identify the issue.
> > >
> > > Solr Version: 8.11.1
> > > Instances: Single instance per machine. No other softwares running on
> the
> > > machine.
> > > CPU: 6 CPU's
> > > Heap: 30GB per Instance
> > > RAM: 60 GB
> > >
> > > --
> > >
> > > Thanks and Regards,
> > > Hari
> > > Mobile:9790756568
> > >
> >
>
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
--
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!
Re: Solr Node High CPU
Posted by Markus Jelsma <ma...@openindex.io>.
You can check the logs to see what it is doing. If there is nothing obvious
in the logs, use a tool like VisualVM to inspect the JVM and perform CPU
sampling. With it, you can see which parts of the JVM are active and eating
CPU.
Regards,
Markus
Op vr 24 mrt 2023 om 12:41 schreef HariBabu kuruva <
hari2708.kuruva@gmail.com>:
> It's the solr process consuming the 95% CPU. I want to know how to
> troubleshoot this from the solr end.
>
> On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com> wrote:
>
> > can you check which process is eating up the CPU using top command
> >
> > On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <ha...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > We have 4 solr nodes, only one of the nodes is consuming almost High
> CPU
> > > from yesterday. We don't see any errors in the logs. Restart dint
> resolve
> > > the issue. Could you please help me to identify the issue.
> > >
> > > Solr Version: 8.11.1
> > > Instances: Single instance per machine. No other softwares running on
> the
> > > machine.
> > > CPU: 6 CPU's
> > > Heap: 30GB per Instance
> > > RAM: 60 GB
> > >
> > > --
> > >
> > > Thanks and Regards,
> > > Hari
> > > Mobile:9790756568
> > >
> >
>
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>
Re: Solr Node High CPU
Posted by HariBabu kuruva <ha...@gmail.com>.
It's the solr process consuming the 95% CPU. I want to know how to
troubleshoot this from the solr end.
On Fri, Mar 24, 2023 at 5:03 PM Deepak Goel <de...@gmail.com> wrote:
> can you check which process is eating up the CPU using top command
>
> On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <ha...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > We have 4 solr nodes, only one of the nodes is consuming almost High CPU
> > from yesterday. We don't see any errors in the logs. Restart dint resolve
> > the issue. Could you please help me to identify the issue.
> >
> > Solr Version: 8.11.1
> > Instances: Single instance per machine. No other softwares running on the
> > machine.
> > CPU: 6 CPU's
> > Heap: 30GB per Instance
> > RAM: 60 GB
> >
> > --
> >
> > Thanks and Regards,
> > Hari
> > Mobile:9790756568
> >
>
--
Thanks and Regards,
Hari
Mobile:9790756568
Re: Solr Node High CPU
Posted by Deepak Goel <de...@gmail.com>.
can you check which process is eating up the CPU using top command
On Fri, 24 Mar 2023, 15:54 HariBabu kuruva, <ha...@gmail.com>
wrote:
> Hi All,
>
> We have 4 solr nodes, only one of the nodes is consuming almost High CPU
> from yesterday. We don't see any errors in the logs. Restart dint resolve
> the issue. Could you please help me to identify the issue.
>
> Solr Version: 8.11.1
> Instances: Single instance per machine. No other softwares running on the
> machine.
> CPU: 6 CPU's
> Heap: 30GB per Instance
> RAM: 60 GB
>
> --
>
> Thanks and Regards,
> Hari
> Mobile:9790756568
>