You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Eric Robinson <er...@psmnv.com> on 2019/08/08 16:53:11 UTC
Tomcat Server Using 100% CPU
We have a farm of VMs, each running multiple instances of tomcat (up to 80 instances per server). Everything has been running fine for years, but recently one server has started nailing the CPU to 100% utilization.
We have tried:
* Different versions of tomcat and JDK
* Doubling the resources to 16 cores and 56 GB RAM
* Moving the VM to different physical server
* Rebuilding the tomcat instances on a brand new VM using Windows Server 2019
* Rebuilding the tomcat instances on a brand new VM using Red Hat Enterprise Linux 7.5
Nothing has worked. No matter where we run the tomcats, they drive CPU up to 100%. Meanwhile the other six servers are still running fine. They all run the same canned tomcat applications.
We would appreciate some guidance on getting to the bottom of this problem.
--Eric
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
RE: Tomcat Server Using 100% CPU
Posted by Eric Robinson <er...@psmnv.com>.
Paul --
I've only used jconsole and thread dumps before. Never used jstack. I'll look into it.
On Thu, 08 Aug 2019, 20:22 Coty Sutherland, <cs...@apache.org> wrote:
> I'd suggest writing a small script to loop about 10 times and capture
> top and thread dumps with jstack at the same time, then wait a few
> seconds then repeat. After that you can grab the pid/tid from the top
> output and compare that with your thread dump to see exactly what the
> thread is doing for the iteration/duration you specify.
>
> Other questions that I haven't seen asked, how long does the CPU usage
> persist? Is it only at startup or does it randomly start after some uptime?
> Have your webapps or dependencies changed around the time the issue
> started? Do the working and nonworking servers run the same webapps
> with the same workload?
>
> On Thu, Aug 8, 2019 at 2:09 PM Eric Robinson <er...@psmnv.com>
> wrote:
>
> > Utkarsh and John, thank you for your feedback.
> >
> > Since everything was originally on Windows, and we built a new Linux
> > server with fresh tomcat installs, and the only thing we moved over
> > from the old Windows servers was the tomcat application folder
> > itself, and the 100% CPU problem persisted, I can't imagine what
> > else could be causing it except the tomcats, but I'm open to ideas.
> >
> > When it happens, all the tomcats start using high CPU at the same time.
> > See the following top output.
> >
> > top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85,
> > 32.67,
> > 34.89
> > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> > %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7
> > si,
> > 0.0 st
> > KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464
> buff/cache
> > KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> > 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> > 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> > 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> > 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> > 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> > 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> > 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> > 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> > 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> > 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> > 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> > 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> > 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> > 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15
> > ksoftirqd/7
> > 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> > 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31
> > kworker/7:0
> >
> > I'll check the localhost_access logs and see if something suspicious
> > stands out.
> >
> > --Eric
> >
> >
> > -----Original Message-----
> > From: Utkarsh Dave <ut...@gmail.com>
> > Sent: Thursday, August 8, 2019 12:33 PM
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Subject: Re: Tomcat Server Using 100% CPU
> >
> > Did you reviewed the localhost_access log file. Which
> > web-application is using tomcat the most ?
> >
> > On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson
> > <er...@psmnv.com>
> > wrote:
> >
> > > We have a farm of VMs, each running multiple instances of tomcat
> > > (up to 80 instances per server). Everything has been running fine
> > > for years, but recently one server has started nailing the CPU to
> > > 100%
> > utilization.
> > >
> > > We have tried:
> > >
> > >
> > > * Different versions of tomcat and JDK
> > > * Doubling the resources to 16 cores and 56 GB RAM
> > > * Moving the VM to different physical server
> > > * Rebuilding the tomcat instances on a brand new VM using Windows
> > > Server 2019
> > > * Rebuilding the tomcat instances on a brand new VM using Red Hat
> > > Enterprise Linux 7.5
> > >
> > > Nothing has worked. No matter where we run the tomcats, they drive
> > > CPU up to 100%. Meanwhile the other six servers are still running fine.
> > > They all run the same canned tomcat applications.
> > >
> > > We would appreciate some guidance on getting to the bottom of this
> > problem.
> > >
> > > --Eric
> > >
> > >
> > > Disclaimer : This email and any files transmitted with it are
> > > confidential and intended solely for intended recipients. If you
> > > are not the named addressee you should not disseminate,
> > > distribute, copy or
> > alter this email.
> > > Any views or opinions presented in this email are solely those of
> > > the author and might not represent those of Physician Select Management.
> > > Warning: Although Physician Select Management has taken reasonable
> > > precautions to ensure no viruses are present in this email, the
> > > company cannot accept responsibility for any loss or damage
> > > arising from the use of this email or attachments.
> > >
> > Disclaimer : This email and any files transmitted with it are
> confidential
> > and intended solely for intended recipients. If you are not the
> > named addressee you should not disseminate, distribute, copy or
> > alter this
> email.
> > Any views or opinions presented in this email are solely those of
> > the author and might not represent those of Physician Select Management.
> > Warning: Although Physician Select Management has taken reasonable
> > precautions to ensure no viruses are present in this email, the
> > company cannot accept responsibility for any loss or damage arising
> > from the use
> of
> > this email or attachments.
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
Re: Tomcat Server Using 100% CPU
Posted by Paul Carter-Brown <pa...@jini.guru>.
Please post a jstack of the pid 10 seconds apart. That will reveal all.
On Thu, 08 Aug 2019, 20:22 Coty Sutherland, <cs...@apache.org> wrote:
> I'd suggest writing a small script to loop about 10 times and capture top
> and thread dumps with jstack at the same time, then wait a few seconds then
> repeat. After that you can grab the pid/tid from the top output and compare
> that with your thread dump to see exactly what the thread is doing for the
> iteration/duration you specify.
>
> Other questions that I haven't seen asked, how long does the CPU usage
> persist? Is it only at startup or does it randomly start after some uptime?
> Have your webapps or dependencies changed around the time the issue
> started? Do the working and nonworking servers run the same webapps with
> the same workload?
>
> On Thu, Aug 8, 2019 at 2:09 PM Eric Robinson <er...@psmnv.com>
> wrote:
>
> > Utkarsh and John, thank you for your feedback.
> >
> > Since everything was originally on Windows, and we built a new Linux
> > server with fresh tomcat installs, and the only thing we moved over from
> > the old Windows servers was the tomcat application folder itself, and the
> > 100% CPU problem persisted, I can't imagine what else could be causing it
> > except the tomcats, but I'm open to ideas.
> >
> > When it happens, all the tomcats start using high CPU at the same time.
> > See the following top output.
> >
> > top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67,
> > 34.89
> > Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> > %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si,
> > 0.0 st
> > KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464
> buff/cache
> > KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail
> Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> > 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> > 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> > 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> > 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> > 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> > 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> > 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> > 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> > 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> > 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> > 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> > 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> > 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> > 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15
> > ksoftirqd/7
> > 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> > 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31
> > kworker/7:0
> >
> > I'll check the localhost_access logs and see if something suspicious
> > stands out.
> >
> > --Eric
> >
> >
> > -----Original Message-----
> > From: Utkarsh Dave <ut...@gmail.com>
> > Sent: Thursday, August 8, 2019 12:33 PM
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Subject: Re: Tomcat Server Using 100% CPU
> >
> > Did you reviewed the localhost_access log file. Which web-application is
> > using tomcat the most ?
> >
> > On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> > wrote:
> >
> > > We have a farm of VMs, each running multiple instances of tomcat (up
> > > to 80 instances per server). Everything has been running fine for
> > > years, but recently one server has started nailing the CPU to 100%
> > utilization.
> > >
> > > We have tried:
> > >
> > >
> > > * Different versions of tomcat and JDK
> > > * Doubling the resources to 16 cores and 56 GB RAM
> > > * Moving the VM to different physical server
> > > * Rebuilding the tomcat instances on a brand new VM using Windows
> > > Server 2019
> > > * Rebuilding the tomcat instances on a brand new VM using Red Hat
> > > Enterprise Linux 7.5
> > >
> > > Nothing has worked. No matter where we run the tomcats, they drive CPU
> > > up to 100%. Meanwhile the other six servers are still running fine.
> > > They all run the same canned tomcat applications.
> > >
> > > We would appreciate some guidance on getting to the bottom of this
> > problem.
> > >
> > > --Eric
> > >
> > >
> > > Disclaimer : This email and any files transmitted with it are
> > > confidential and intended solely for intended recipients. If you are
> > > not the named addressee you should not disseminate, distribute, copy or
> > alter this email.
> > > Any views or opinions presented in this email are solely those of the
> > > author and might not represent those of Physician Select Management.
> > > Warning: Although Physician Select Management has taken reasonable
> > > precautions to ensure no viruses are present in this email, the
> > > company cannot accept responsibility for any loss or damage arising
> > > from the use of this email or attachments.
> > >
> > Disclaimer : This email and any files transmitted with it are
> confidential
> > and intended solely for intended recipients. If you are not the named
> > addressee you should not disseminate, distribute, copy or alter this
> email.
> > Any views or opinions presented in this email are solely those of the
> > author and might not represent those of Physician Select Management.
> > Warning: Although Physician Select Management has taken reasonable
> > precautions to ensure no viruses are present in this email, the company
> > cannot accept responsibility for any loss or damage arising from the use
> of
> > this email or attachments.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
RE: Tomcat Server Using 100% CPU
Posted by Eric Robinson <er...@psmnv.com>.
Coty --
There is a normal period of high CPU right after startup, but that's expected. However, it usually peaks well after startup (minutes or hours) and then it stays that way, sometimes all night. When it happens, all the tomcats (about 20 of them) running show high CPU, like 20-50% each.
I can write a script to loop and output top and thread dumps. Should I do it for all running instances or just for a selected target instance?
--Eric
-----Original Message-----
From: Coty Sutherland <cs...@apache.org>
Sent: Thursday, August 8, 2019 1:22 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Tomcat Server Using 100% CPU
I'd suggest writing a small script to loop about 10 times and capture top and thread dumps with jstack at the same time, then wait a few seconds then repeat. After that you can grab the pid/tid from the top output and compare that with your thread dump to see exactly what the thread is doing for the iteration/duration you specify.
Other questions that I haven't seen asked, how long does the CPU usage persist? Is it only at startup or does it randomly start after some uptime?
Have your webapps or dependencies changed around the time the issue started? Do the working and nonworking servers run the same webapps with the same workload?
On Thu, Aug 8, 2019 at 2:09 PM Eric Robinson <er...@psmnv.com>
wrote:
> Utkarsh and John, thank you for your feedback.
>
> Since everything was originally on Windows, and we built a new Linux
> server with fresh tomcat installs, and the only thing we moved over
> from the old Windows servers was the tomcat application folder itself,
> and the 100% CPU problem persisted, I can't imagine what else could be
> causing it except the tomcats, but I'm open to ideas.
>
> When it happens, all the tomcats start using high CPU at the same time.
> See the following top output.
>
> top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67,
> 34.89
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7
> si,
> 0.0 st
> KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
> KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15
> ksoftirqd/7
> 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31
> kworker/7:0
>
> I'll check the localhost_access logs and see if something suspicious
> stands out.
>
> --Eric
>
>
> -----Original Message-----
> From: Utkarsh Dave <ut...@gmail.com>
> Sent: Thursday, August 8, 2019 12:33 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Server Using 100% CPU
>
> Did you reviewed the localhost_access log file. Which web-application
> is using tomcat the most ?
>
> On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> wrote:
>
> > We have a farm of VMs, each running multiple instances of tomcat (up
> > to 80 instances per server). Everything has been running fine for
> > years, but recently one server has started nailing the CPU to 100%
> utilization.
> >
> > We have tried:
> >
> >
> > * Different versions of tomcat and JDK
> > * Doubling the resources to 16 cores and 56 GB RAM
> > * Moving the VM to different physical server
> > * Rebuilding the tomcat instances on a brand new VM using Windows
> > Server 2019
> > * Rebuilding the tomcat instances on a brand new VM using Red Hat
> > Enterprise Linux 7.5
> >
> > Nothing has worked. No matter where we run the tomcats, they drive
> > CPU up to 100%. Meanwhile the other six servers are still running fine.
> > They all run the same canned tomcat applications.
> >
> > We would appreciate some guidance on getting to the bottom of this
> problem.
> >
> > --Eric
> >
> >
> > Disclaimer : This email and any files transmitted with it are
> > confidential and intended solely for intended recipients. If you are
> > not the named addressee you should not disseminate, distribute, copy
> > or
> alter this email.
> > Any views or opinions presented in this email are solely those of
> > the author and might not represent those of Physician Select Management.
> > Warning: Although Physician Select Management has taken reasonable
> > precautions to ensure no viruses are present in this email, the
> > company cannot accept responsibility for any loss or damage arising
> > from the use of this email or attachments.
> >
> Disclaimer : This email and any files transmitted with it are
> confidential and intended solely for intended recipients. If you are
> not the named addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the
> company cannot accept responsibility for any loss or damage arising
> from the use of this email or attachments.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Server Using 100% CPU
Posted by Coty Sutherland <cs...@apache.org>.
I'd suggest writing a small script to loop about 10 times and capture top
and thread dumps with jstack at the same time, then wait a few seconds then
repeat. After that you can grab the pid/tid from the top output and compare
that with your thread dump to see exactly what the thread is doing for the
iteration/duration you specify.
Other questions that I haven't seen asked, how long does the CPU usage
persist? Is it only at startup or does it randomly start after some uptime?
Have your webapps or dependencies changed around the time the issue
started? Do the working and nonworking servers run the same webapps with
the same workload?
On Thu, Aug 8, 2019 at 2:09 PM Eric Robinson <er...@psmnv.com>
wrote:
> Utkarsh and John, thank you for your feedback.
>
> Since everything was originally on Windows, and we built a new Linux
> server with fresh tomcat installs, and the only thing we moved over from
> the old Windows servers was the tomcat application folder itself, and the
> 100% CPU problem persisted, I can't imagine what else could be causing it
> except the tomcats, but I'm open to ideas.
>
> When it happens, all the tomcats start using high CPU at the same time.
> See the following top output.
>
> top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67,
> 34.89
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si,
> 0.0 st
> KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
> KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15
> ksoftirqd/7
> 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31
> kworker/7:0
>
> I'll check the localhost_access logs and see if something suspicious
> stands out.
>
> --Eric
>
>
> -----Original Message-----
> From: Utkarsh Dave <ut...@gmail.com>
> Sent: Thursday, August 8, 2019 12:33 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Server Using 100% CPU
>
> Did you reviewed the localhost_access log file. Which web-application is
> using tomcat the most ?
>
> On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> wrote:
>
> > We have a farm of VMs, each running multiple instances of tomcat (up
> > to 80 instances per server). Everything has been running fine for
> > years, but recently one server has started nailing the CPU to 100%
> utilization.
> >
> > We have tried:
> >
> >
> > * Different versions of tomcat and JDK
> > * Doubling the resources to 16 cores and 56 GB RAM
> > * Moving the VM to different physical server
> > * Rebuilding the tomcat instances on a brand new VM using Windows
> > Server 2019
> > * Rebuilding the tomcat instances on a brand new VM using Red Hat
> > Enterprise Linux 7.5
> >
> > Nothing has worked. No matter where we run the tomcats, they drive CPU
> > up to 100%. Meanwhile the other six servers are still running fine.
> > They all run the same canned tomcat applications.
> >
> > We would appreciate some guidance on getting to the bottom of this
> problem.
> >
> > --Eric
> >
> >
> > Disclaimer : This email and any files transmitted with it are
> > confidential and intended solely for intended recipients. If you are
> > not the named addressee you should not disseminate, distribute, copy or
> alter this email.
> > Any views or opinions presented in this email are solely those of the
> > author and might not represent those of Physician Select Management.
> > Warning: Although Physician Select Management has taken reasonable
> > precautions to ensure no viruses are present in this email, the
> > company cannot accept responsibility for any loss or damage arising
> > from the use of this email or attachments.
> >
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
RE: Tomcat Server Using 100% CPU
Posted by Eric Robinson <er...@psmnv.com>.
André, Paul, and Coty, you've all provided some great next steps. I'll investigate further and be back in touch!
--Eric
-----Original Message-----
From: André Warnier (tomcat) <aw...@ice-sa.com>
Sent: Thursday, August 8, 2019 3:53 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat Server Using 100% CPU
On 08.08.2019 20:08, Eric Robinson wrote:
> Utkarsh and John, thank you for your feedback.
>
> Since everything was originally on Windows, and we built a new Linux server with fresh tomcat installs, and the only thing we moved over from the old Windows servers was the tomcat application folder itself, and the 100% CPU problem persisted, I can't imagine what else could be causing it except the tomcats, but I'm open to ideas.
>
> When it happens, all the tomcats start using high CPU at the same time. See the following top output.
>
> top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67, 34.89
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7
> si, 0.0 st KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
> KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15 ksoftirqd/7
> 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31 kworker/7:0
>
> I'll check the localhost_access logs and see if something suspicious stands out.
>
Access logs is the first thing to look at of course (just in case you are subject to some DoS attack), but other things of interest :
1) what is "the webapp" in question ? Any reason to suspect it may have been been hijacked, to do something it is not supposed to do ? does the webapp allow for clients to upload "things" to the server (files, documents, images,..) ?
2) if you look at the tomcat logs, do you recognise all the webapps which get deployed when tomcat starts ? Or is there an alien there ?
3) if you run "top" again, then enter a "c" in the console, it will show more details about the "java" command it is running.
Similarly, doing a "ps -ef" command and comparing the result (by PID) with the top output, may give more details.
That would show (us) at least the startup parameters of your tomcat(s).
4) speaking as a faithful Tomcat Committer, we always like to repeat that in 99% of the cases it turns out that the problem is with the webapp, not with Tomcat. The fact that in your case, you have changed about everything except the webapp, and the problem persists, would only tend to increase that suspicion..
So, can the webapp do any logging that would show what it's doing, while it is happily slurping all your CPU time ?
6) does the tomcat error log show anything of interest ?
7) under Linux as root, enter : netstat --tcp -pan | grep LISTEN (Shows all TCP ports your server is listening to, and which PID/processes control these ports).
Anything unexpected there ? worse : anything unexpected which would match the PID of one of your tomcats ?
André
> --Eric
>
>
> -----Original Message-----
> From: Utkarsh Dave <ut...@gmail.com>
> Sent: Thursday, August 8, 2019 12:33 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Server Using 100% CPU
>
> Did you reviewed the localhost_access log file. Which web-application is using tomcat the most ?
>
> On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> wrote:
>
>> We have a farm of VMs, each running multiple instances of tomcat (up
>> to 80 instances per server). Everything has been running fine for
>> years, but recently one server has started nailing the CPU to 100% utilization.
>>
>> We have tried:
>>
>>
>> * Different versions of tomcat and JDK
>> * Doubling the resources to 16 cores and 56 GB RAM
>> * Moving the VM to different physical server
>> * Rebuilding the tomcat instances on a brand new VM using Windows
>> Server 2019
>> * Rebuilding the tomcat instances on a brand new VM using Red Hat
>> Enterprise Linux 7.5
>>
>> Nothing has worked. No matter where we run the tomcats, they drive
>> CPU up to 100%. Meanwhile the other six servers are still running fine.
>> They all run the same canned tomcat applications.
>>
>> We would appreciate some guidance on getting to the bottom of this problem.
>>
>> --Eric
>>
>>
>> Disclaimer : This email and any files transmitted with it are
>> confidential and intended solely for intended recipients. If you are
>> not the named addressee you should not disseminate, distribute, copy or alter this email.
>> Any views or opinions presented in this email are solely those of the
>> author and might not represent those of Physician Select Management.
>> Warning: Although Physician Select Management has taken reasonable
>> precautions to ensure no viruses are present in this email, the
>> company cannot accept responsibility for any loss or damage arising
>> from the use of this email or attachments.
>>
> Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Server Using 100% CPU
Posted by Eric Robinson <er...@psmnv.com>.
André --
To answer your questions...
1) what is "the webapp" in question ? Any reason to suspect it may have been been hijacked, to do something it is not supposed to do ? does the webapp allow for clients to upload "things" to the server (files, documents, images,..) ?
It's a canned commercial application, only accessible from within the private network. Nothing public-facing.
2) if you look at the tomcat logs, do you recognise all the webapps which get deployed when tomcat starts ? Or is there an alien there ?
Yes, I recognize them all as components of the application.
3) if you run "top" again, then enter a "c" in the console, it will show more details about the "java" command it is running.
Similarly, doing a "ps -ef" command and comparing the result (by PID) with the top output, may give more details.
That would show (us) at least the startup parameters of your tomcat(s).
There are currently 25 instances of tomcat, all started with the same basic parameters (except referencing different directories). Here is an example.
site522 31047 1 23 04:42 ? 00:44:00 /usr/java/jdk1.7.0_79/bin/java -Djava.util.logging.config.file=/alley/site522/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms128M -Xmx448M -XX:MaxPermSize=192M -Djvm=tomcat7_522 -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Duser.timezone=US/Central -Dhttps.protocols=TLSv1.1,TLSv1.2 -Xloggc:/alley/site522/tomcat7/logs/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Djava.rmi.server.hostname=lab03 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=6522 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.password.file=/alley/site522/tomcat7/conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/alley/site522/tomcat7/conf/jmxremote.access -Djdk.tls.ephemeralDHKeySize=2048 -Djava.endorsed.dirs=/alley/site522/tomcat7/endorsed -classpath /alley/site522/tomcat7/bin/bootstrap.jar:/alley/site522/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/alley/site522/tomcat7 -Dcatalina.home=/alley/site522/tomcat7 -Djava.io.tmpdir=/alley/site522/tomcat7/temp org.apache.catalina.startup.Bootstrap start
4) speaking as a faithful Tomcat Committer, we always like to repeat that in 99% of the cases it turns out that the problem is with the webapp, not with Tomcat. The fact that in your case, you have changed about everything except the webapp, and the problem persists, would only tend to increase that suspicion..
That's my suspicion, too. The only problem is that the vendor has book looking into it and has not found the cause, so now I'm trying to help the vendor out by appealing to the wider community.
So, can the webapp do any logging that would show what it's doing, while it is happily slurping all your CPU time ?
6) does the tomcat error log show anything of interest ?
I see a lot of these... (note that the app names have been changed for security purposes).
SEVERE: Failed to initialize end point associated with ProtocolHandler ["ajp-bio-8009"]
SEVERE: Failed to initialize connector [Connector[AJP/1.3-8009]]
SEVERE: Failed to initialize end point associated with ProtocolHandler ["ajp-bio-8009"]
SEVERE: Failed to initialize connector [Connector[AJP/1.3-8009]]
SEVERE: Failed to initialize end point associated with ProtocolHandler ["ajp-bio-8009"]
SEVERE: Failed to initialize connector [Connector[AJP/1.3-8009]]
SEVERE: The web application [/moby] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: The web application [/moby] appears to have started a thread named [MySQL Statement Cancellation Timer] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [Thread-8] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [Thread-10] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [Timer-3] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [Selector Worker: 0] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [ActiveMQ InactivityMonitor ReadCheckTimer] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [ActiveMQ InactivityMonitor WriteCheckTimer] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [Timer-5] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [ActiveMQ NIO Worker 3] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/moby] appears to have started a thread named [ActiveMQ InactivityMonitor Worker] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/speaky] registered the JDBC driver [com.ecw.jdbc.EcwDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: The web application [/speaky] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
SEVERE: Failed to initialize end point associated with ProtocolHandler ["ajp-bio-8009"]
SEVERE: Failed to initialize connector [Connector[AJP/1.3-8009]]
7) under Linux as root, enter : netstat --tcp -pan | grep LISTEN (Shows all TCP ports your server is listening to, and which PID/processes control these ports).
Anything unexpected there ? worse : anything unexpected which would match the PID of one of your tomcats ?
I did notice some unexpected ports being used by java.
Here is output from one of our other tomcat servers that is working fine. In fact, all the other tomcat servers look like this. The listening port is always 4 digits and starts with a 3, by design.
tcp 0 0 0.0.0.0:3763 0.0.0.0:* LISTEN 19259/java
tcp 0 0 0.0.0.0:3731 0.0.0.0:* LISTEN 4189/java
tcp 0 0 0.0.0.0:3539 0.0.0.0:* LISTEN 18253/java
tcp 0 0 0.0.0.0:3764 0.0.0.0:* LISTEN 21103/java
tcp 0 0 0.0.0.0:3700 0.0.0.0:* LISTEN 24590/java
tcp 0 0 0.0.0.0:3765 0.0.0.0:* LISTEN 23328/java
tcp 0 0 0.0.0.0:3733 0.0.0.0:* LISTEN 5909/java
tcp 0 0 0.0.0.0:3701 0.0.0.0:* LISTEN 30779/java
tcp 0 0 0.0.0.0:3766 0.0.0.0:* LISTEN 25008/java
tcp 0 0 0.0.0.0:3734 0.0.0.0:* LISTEN 6944/java
tcp 0 0 0.0.0.0:3062 0.0.0.0:* LISTEN 12697/java
tcp 0 0 0.0.0.0:3767 0.0.0.0:* LISTEN 26636/java
tcp 0 0 0.0.0.0:3703 0.0.0.0:* LISTEN 309/java
tcp 0 0 0.0.0.0:3768 0.0.0.0:* LISTEN 28688/java
tcp 0 0 0.0.0.0:3480 0.0.0.0:* LISTEN 16094/java
tcp 0 0 0.0.0.0:3320 0.0.0.0:* LISTEN 24815/java
tcp 0 0 0.0.0.0:3769 0.0.0.0:* LISTEN 30529/java
tcp 0 0 0.0.0.0:3737 0.0.0.0:* LISTEN 9767/java
tcp 0 0 0.0.0.0:3705 0.0.0.0:* LISTEN 2336/java
tcp 0 0 0.0.0.0:3065 0.0.0.0:* LISTEN 14284/java
tcp 0 0 0.0.0.0:3770 0.0.0.0:* LISTEN 32347/java
tcp 0 0 0.0.0.0:3706 0.0.0.0:* LISTEN 4955/java
tcp 0 0 0.0.0.0:3098 0.0.0.0:* LISTEN 17623/java
But on the server in question, I see a bunch of other ports that I don’t recognize. I don't know where the ones with 5 digits are coming from. (The 4-digit ones starting with 6 are the JMX monitoring ports I set up, so those are expected.)
tcp 0 0 0.0.0.0:6535 0.0.0.0:* LISTEN 31566/java
tcp 0 0 0.0.0.0:38919 0.0.0.0:* LISTEN 30166/java
tcp 0 0 0.0.0.0:40425 0.0.0.0:* LISTEN 6849/java
tcp 0 0 0.0.0.0:6473 0.0.0.0:* LISTEN 6849/java
tcp 0 0 0.0.0.0:35785 0.0.0.0:* LISTEN 17313/java
tcp 0 0 0.0.0.0:40009 0.0.0.0:* LISTEN 19622/java
tcp 0 0 0.0.0.0:44265 0.0.0.0:* LISTEN 30370/java
tcp 0 0 0.0.0.0:36329 0.0.0.0:* LISTEN 30166/java
tcp 0 0 0.0.0.0:42922 0.0.0.0:* LISTEN 8597/java
tcp 0 0 0.0.0.0:3434 0.0.0.0:* LISTEN 17313/java
tcp 0 0 0.0.0.0:6506 0.0.0.0:* LISTEN 4447/java
tcp 0 0 0.0.0.0:36778 0.0.0.0:* LISTEN 300/java
tcp 0 0 0.0.0.0:6475 0.0.0.0:* LISTEN 7261/java
tcp 0 0 0.0.0.0:35787 0.0.0.0:* LISTEN 19622/java
tcp 0 0 0.0.0.0:46795 0.0.0.0:* LISTEN 3673/java
tcp 0 0 0.0.0.0:3211 0.0.0.0:* LISTEN 30166/java
tcp 0 0 0.0.0.0:3436 0.0.0.0:* LISTEN 31863/java
tcp 0 0 0.0.0.0:3116 0.0.0.0:* LISTEN 29948/java
tcp 0 0 0.0.0.0:6509 0.0.0.0:* LISTEN 9462/java
tcp 0 0 0.0.0.0:6477 0.0.0.0:* LISTEN 7678/java
tcp 0 0 0.0.0.0:39885 0.0.0.0:* LISTEN 4447/java
tcp 0 0 0.0.0.0:33581 0.0.0.0:* LISTEN 300/java
tcp 0 0 0.0.0.0:6478 0.0.0.0:* LISTEN 8027/java
-----Original Message-----
From: André Warnier (tomcat) <aw...@ice-sa.com>
Sent: Thursday, August 8, 2019 3:53 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat Server Using 100% CPU
On 08.08.2019 20:08, Eric Robinson wrote:
> Utkarsh and John, thank you for your feedback.
>
> Since everything was originally on Windows, and we built a new Linux server with fresh tomcat installs, and the only thing we moved over from the old Windows servers was the tomcat application folder itself, and the 100% CPU problem persisted, I can't imagine what else could be causing it except the tomcats, but I'm open to ideas.
>
> When it happens, all the tomcats start using high CPU at the same time. See the following top output.
>
> top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67, 34.89
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7
> si, 0.0 st KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
> KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15 ksoftirqd/7
> 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31 kworker/7:0
>
> I'll check the localhost_access logs and see if something suspicious stands out.
>
Access logs is the first thing to look at of course (just in case you are subject to some DoS attack), but other things of interest :
1) what is "the webapp" in question ? Any reason to suspect it may have been been hijacked, to do something it is not supposed to do ? does the webapp allow for clients to upload "things" to the server (files, documents, images,..) ?
2) if you look at the tomcat logs, do you recognise all the webapps which get deployed when tomcat starts ? Or is there an alien there ?
3) if you run "top" again, then enter a "c" in the console, it will show more details about the "java" command it is running.
Similarly, doing a "ps -ef" command and comparing the result (by PID) with the top output, may give more details.
That would show (us) at least the startup parameters of your tomcat(s).
4) speaking as a faithful Tomcat Committer, we always like to repeat that in 99% of the cases it turns out that the problem is with the webapp, not with Tomcat. The fact that in your case, you have changed about everything except the webapp, and the problem persists, would only tend to increase that suspicion..
So, can the webapp do any logging that would show what it's doing, while it is happily slurping all your CPU time ?
6) does the tomcat error log show anything of interest ?
7) under Linux as root, enter : netstat --tcp -pan | grep LISTEN (Shows all TCP ports your server is listening to, and which PID/processes control these ports).
Anything unexpected there ? worse : anything unexpected which would match the PID of one of your tomcats ?
André
> --Eric
>
>
> -----Original Message-----
> From: Utkarsh Dave <ut...@gmail.com>
> Sent: Thursday, August 8, 2019 12:33 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Server Using 100% CPU
>
> Did you reviewed the localhost_access log file. Which web-application is using tomcat the most ?
>
> On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> wrote:
>
>> We have a farm of VMs, each running multiple instances of tomcat (up
>> to 80 instances per server). Everything has been running fine for
>> years, but recently one server has started nailing the CPU to 100% utilization.
>>
>> We have tried:
>>
>>
>> * Different versions of tomcat and JDK
>> * Doubling the resources to 16 cores and 56 GB RAM
>> * Moving the VM to different physical server
>> * Rebuilding the tomcat instances on a brand new VM using Windows
>> Server 2019
>> * Rebuilding the tomcat instances on a brand new VM using Red Hat
>> Enterprise Linux 7.5
>>
>> Nothing has worked. No matter where we run the tomcats, they drive
>> CPU up to 100%. Meanwhile the other six servers are still running fine.
>> They all run the same canned tomcat applications.
>>
>> We would appreciate some guidance on getting to the bottom of this problem.
>>
>> --Eric
>>
>>
>> Disclaimer : This email and any files transmitted with it are
>> confidential and intended solely for intended recipients. If you are
>> not the named addressee you should not disseminate, distribute, copy or alter this email.
>> Any views or opinions presented in this email are solely those of the
>> author and might not represent those of Physician Select Management.
>> Warning: Although Physician Select Management has taken reasonable
>> precautions to ensure no viruses are present in this email, the
>> company cannot accept responsibility for any loss or damage arising
>> from the use of this email or attachments.
>>
> Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
Re: Tomcat Server Using 100% CPU
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 08.08.2019 20:08, Eric Robinson wrote:
> Utkarsh and John, thank you for your feedback.
>
> Since everything was originally on Windows, and we built a new Linux server with fresh tomcat installs, and the only thing we moved over from the old Windows servers was the tomcat application folder itself, and the 100% CPU problem persisted, I can't imagine what else could be causing it except the tomcats, but I'm open to ideas.
>
> When it happens, all the tomcats start using high CPU at the same time. See the following top output.
>
> top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67, 34.89
> Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
> KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
> KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
> 20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
> 21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
> 20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
> 19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
> 19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
> 19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
> 20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
> 20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
> 20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
> 20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
> 19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
> 20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
> 19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
> 44 root 20 0 0 0 0 R 19.1 0.0 159:46.15 ksoftirqd/7
> 3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
> 2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31 kworker/7:0
>
> I'll check the localhost_access logs and see if something suspicious stands out.
>
Access logs is the first thing to look at of course (just in case you are subject to some
DoS attack), but other things of interest :
1) what is "the webapp" in question ? Any reason to suspect it may have been been
hijacked, to do something it is not supposed to do ? does the webapp allow for clients to
upload "things" to the server (files, documents, images,..) ?
2) if you look at the tomcat logs, do you recognise all the webapps which get deployed
when tomcat starts ? Or is there an alien there ?
3) if you run "top" again, then enter a "c" in the console, it will show more details
about the "java" command it is running.
Similarly, doing a "ps -ef" command and comparing the result (by PID) with the top output,
may give more details.
That would show (us) at least the startup parameters of your tomcat(s).
4) speaking as a faithful Tomcat Committer, we always like to repeat that in 99% of the
cases it turns out that the problem is with the webapp, not with Tomcat. The fact that in
your case, you have changed about everything except the webapp, and the problem persists,
would only tend to increase that suspicion..
So, can the webapp do any logging that would show what it's doing, while it is happily
slurping all your CPU time ?
6) does the tomcat error log show anything of interest ?
7) under Linux as root, enter : netstat --tcp -pan | grep LISTEN
(Shows all TCP ports your server is listening to, and which PID/processes control these
ports).
Anything unexpected there ? worse : anything unexpected which would match the PID of one
of your tomcats ?
André
> --Eric
>
>
> -----Original Message-----
> From: Utkarsh Dave <ut...@gmail.com>
> Sent: Thursday, August 8, 2019 12:33 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Re: Tomcat Server Using 100% CPU
>
> Did you reviewed the localhost_access log file. Which web-application is using tomcat the most ?
>
> On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
> wrote:
>
>> We have a farm of VMs, each running multiple instances of tomcat (up
>> to 80 instances per server). Everything has been running fine for
>> years, but recently one server has started nailing the CPU to 100% utilization.
>>
>> We have tried:
>>
>>
>> * Different versions of tomcat and JDK
>> * Doubling the resources to 16 cores and 56 GB RAM
>> * Moving the VM to different physical server
>> * Rebuilding the tomcat instances on a brand new VM using Windows
>> Server 2019
>> * Rebuilding the tomcat instances on a brand new VM using Red Hat
>> Enterprise Linux 7.5
>>
>> Nothing has worked. No matter where we run the tomcats, they drive CPU
>> up to 100%. Meanwhile the other six servers are still running fine.
>> They all run the same canned tomcat applications.
>>
>> We would appreciate some guidance on getting to the bottom of this problem.
>>
>> --Eric
>>
>>
>> Disclaimer : This email and any files transmitted with it are
>> confidential and intended solely for intended recipients. If you are
>> not the named addressee you should not disseminate, distribute, copy or alter this email.
>> Any views or opinions presented in this email are solely those of the
>> author and might not represent those of Physician Select Management.
>> Warning: Although Physician Select Management has taken reasonable
>> precautions to ensure no viruses are present in this email, the
>> company cannot accept responsibility for any loss or damage arising
>> from the use of this email or attachments.
>>
> Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Server Using 100% CPU
Posted by Eric Robinson <er...@psmnv.com>.
Utkarsh and John, thank you for your feedback.
Since everything was originally on Windows, and we built a new Linux server with fresh tomcat installs, and the only thing we moved over from the old Windows servers was the tomcat application folder itself, and the 100% CPU problem persisted, I can't imagine what else could be causing it except the tomcats, but I'm open to ideas.
When it happens, all the tomcats start using high CPU at the same time. See the following top output.
top - 11:06:44 up 1 day, 6:59, 7 users, load average: 36.85, 32.67, 34.89
Tasks: 245 total, 4 running, 241 sleeping, 0 stopped, 0 zombie
%Cpu(s): 80.7 us, 13.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
KiB Mem : 48132572 total, 11677420 free, 5572688 used, 30882464 buff/cache
KiB Swap: 15626236 total, 15584324 free, 41912 used. 41859232 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19379 site211 20 0 3529072 447444 24632 S 120.4 0.9 3:37.19 java
20092 site555 20 0 2530376 375500 24496 S 72.4 0.8 2:01.44 java
21077 site450 20 0 2530292 298260 24292 S 69.6 0.6 1:10.92 java
20378 site436 20 0 3262200 347160 24096 S 68.3 0.7 2:47.26 java
19957 site522 20 0 2596856 373532 24364 S 52.0 0.8 2:37.13 java
19537 site396 20 0 2862724 386860 23820 S 51.1 0.8 2:34.25 java
19228 site116 20 0 3595652 490032 24640 S 50.5 1.0 5:03.28 java
20941 site456 20 0 2596996 338740 24488 S 49.2 0.7 1:32.89 java
20789 site354 20 0 2596920 327612 24480 S 42.9 0.7 1:30.47 java
20657 site327 20 0 3123004 346308 24540 S 41.4 0.7 1:50.97 java
20524 site203 20 0 2458376 340400 25416 S 39.8 0.7 1:48.01 java
19675 site487 20 0 2530296 390948 24408 S 35.7 0.8 2:37.95 java
20233 site535 20 0 2530292 324112 24360 S 32.9 0.7 1:54.31 java
19809 site514 20 0 2530216 400308 24340 S 25.7 0.8 2:35.97 java
44 root 20 0 0 0 0 R 19.1 0.0 159:46.15 ksoftirqd/7
3926 root 20 0 208512 22668 4128 S 16.9 0.0 242:45.07 iotop
2036 root 20 0 0 0 0 R 13.2 0.0 1:38.31 kworker/7:0
I'll check the localhost_access logs and see if something suspicious stands out.
--Eric
-----Original Message-----
From: Utkarsh Dave <ut...@gmail.com>
Sent: Thursday, August 8, 2019 12:33 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Tomcat Server Using 100% CPU
Did you reviewed the localhost_access log file. Which web-application is using tomcat the most ?
On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
wrote:
> We have a farm of VMs, each running multiple instances of tomcat (up
> to 80 instances per server). Everything has been running fine for
> years, but recently one server has started nailing the CPU to 100% utilization.
>
> We have tried:
>
>
> * Different versions of tomcat and JDK
> * Doubling the resources to 16 cores and 56 GB RAM
> * Moving the VM to different physical server
> * Rebuilding the tomcat instances on a brand new VM using Windows
> Server 2019
> * Rebuilding the tomcat instances on a brand new VM using Red Hat
> Enterprise Linux 7.5
>
> Nothing has worked. No matter where we run the tomcats, they drive CPU
> up to 100%. Meanwhile the other six servers are still running fine.
> They all run the same canned tomcat applications.
>
> We would appreciate some guidance on getting to the bottom of this problem.
>
> --Eric
>
>
> Disclaimer : This email and any files transmitted with it are
> confidential and intended solely for intended recipients. If you are
> not the named addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the
> company cannot accept responsibility for any loss or damage arising
> from the use of this email or attachments.
>
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Server Using 100% CPU
Posted by Utkarsh Dave <ut...@gmail.com>.
Did you reviewed the localhost_access log file. Which web-application is
using tomcat the most ?
On Thu, Aug 8, 2019 at 9:53 AM Eric Robinson <er...@psmnv.com>
wrote:
> We have a farm of VMs, each running multiple instances of tomcat (up to 80
> instances per server). Everything has been running fine for years, but
> recently one server has started nailing the CPU to 100% utilization.
>
> We have tried:
>
>
> * Different versions of tomcat and JDK
> * Doubling the resources to 16 cores and 56 GB RAM
> * Moving the VM to different physical server
> * Rebuilding the tomcat instances on a brand new VM using Windows
> Server 2019
> * Rebuilding the tomcat instances on a brand new VM using Red Hat
> Enterprise Linux 7.5
>
> Nothing has worked. No matter where we run the tomcats, they drive CPU up
> to 100%. Meanwhile the other six servers are still running fine. They all
> run the same canned tomcat applications.
>
> We would appreciate some guidance on getting to the bottom of this problem.
>
> --Eric
>
>
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>
RE: Tomcat Server Using 100% CPU
Posted by Jo...@wellsfargo.com.
Eric,
> -----Original Message-----
> From: Eric Robinson <er...@psmnv.com>
> Sent: Thursday, August 08, 2019 11:53 AM
> To: users@tomcat.apache.org
> Subject: Tomcat Server Using 100% CPU
>
> We have a farm of VMs, each running multiple instances of tomcat (up to 80
> instances per server). Everything has been running fine for years, but
> recently one server has started nailing the CPU to 100% utilization.
>
> We have tried:
>
>
> * Different versions of tomcat and JDK
> * Doubling the resources to 16 cores and 56 GB RAM
> * Moving the VM to different physical server
> * Rebuilding the tomcat instances on a brand new VM using Windows
> Server 2019
> * Rebuilding the tomcat instances on a brand new VM using Red Hat
> Enterprise Linux 7.5
>
> Nothing has worked. No matter where we run the tomcats, they drive CPU
> up to 100%. Meanwhile the other six servers are still running fine. They all
> run the same canned tomcat applications.
>
> We would appreciate some guidance on getting to the bottom of this
> problem.
>
> --Eric
>
>
Are you sure that it's the Tomcat process that is using the CPU? If so, take several thread dumps about 5 seconds apart. On Linux, use kill -3 tomcatpidhere. They will go to catalina.out. You'll probably need a developer to interpret them.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org