You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2014/04/03 16:04:28 UTC
[users@httpd] High CPU Usage in Amazon EC2
All,
I'm having a problem in production I've never seem before. We are
running a pair of AWS EC2 m1.micro web servers where only one of them in
really in service at any given time. The httpd instance serves some
static content and forwards a great deal of traffic via stunnel to a
single back-end Tomcat server using mod_jk 1.2.37. We have been running
under this configuration for several years with no problems.
Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
Amazon's previous version (I can't remember which one), including the
package-refresh that comes with it for httpd. The current kernel version
is 3.10.34. The current httpd version is 2.2.26. The package name is
"httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
prefork MPM with the following (default) settings:
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
What I can observe is that the CPU load average is rising from the usual
sub-2.0 value to sometimes as high as 70. That's seventy, not
seven-point-oh.
I see no errors in the log, and httpd doesn't seem to be dropping any
requests... just running very very slowly.
It seems to come in waves: the load will go up, and everything will slow
down, and then we'll get a reprieve.
I can see 22 server processes running right this moment, but the load
average has dropped back to 0.05.
I've enabled ExtendedStatus and it really doesn't look like there is a
huge number of requests being served. Less than 1 req/sec. This is *not*
a high-load server. I can see some of the httpd child processes using
20% or more of the CPU as reported by 'top'.
Is there a good way for me to determine what those processes are doing?
As this is a modestly-used server, I can probably enable additional
logging without too much trouble.
Any help anyone can provide would be very much appreciated.
Thanks,
-chris
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
All,
On 4/8/14, 11:11 AM, Christopher Schultz wrote:
> I'm not sure I can chalk-up the above difference to simply network noise
> (specifically, average, max, 90% line). What I'm observing here is
> certainly not the extent of the problems we are experiencing, but I have
> noticed a difference between these two versions as provided by Amazon's
> yum repo.
I had a flash of inspiration while listening to one of today's talks on TLS.
We recently updated our TLS certificates, and the new server keys are
4096 bits while the older keys were 2048 bits. Of the two servers I
configured for testing, one had the new keys, the other, the old.
Matching the two keys has brought the load test back to a sane comparison.
Now I can resume hunting for the combination of packages and settings
that seems to be the differentiating factor.
-chris
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Igor,
On 4/5/14, 4:57 PM, Igor Cicimov wrote:
>
> On 06/04/2014 5:51 AM, "Christopher Schultz"
> <chris@christopherschultz.net <ma...@christopherschultz.net>> wrote:
>>
>> Igor,
>>
>> On 4/4/14, 5:39 AM, Igor Cicimov wrote:
>> >
>> > On 04/04/2014 1:05 AM, "Christopher Schultz"
>> > <chris@christopherschultz.net <ma...@christopherschultz.net>
> <mailto:chris@christopherschultz.net
> <ma...@christopherschultz.net>>> wrote:
>> >>
>> >> All,
>> >>
>> >> I'm having a problem in production I've never seem before. We are
>> >> running a pair of AWS EC2 m1.micro web servers where only one of
> them in
>> >> really in service at any given time. The httpd instance serves some
>> >> static content and forwards a great deal of traffic via stunnel to a
>> >> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>> >> under this configuration for several years with no problems.
>> >>
>> > Enable the stunnel logs maybe they will reveal something?
>>
>> I don't think stunnel has changed much. Besides, the stunnel processes
>> aren't eating up the CPU: it's the httpd processes that are.
>>
> Yes but apache actually connects to stunnel and can be affected by it in
> case of any issues it has.
>
> I would also try to run pstack again some of the pids consuming the cpu
> to find out what are they doing.
>
> Basically since the kernel got upgraded any of the parts used here might
> be possibly impacted by some new or existing bug.
I have some more data. I have launched two EC2 t1.micro instances based
upon the same old snapshot that I had available. I upgraded 3 packages
on one of them (httpd and two dependencies: mod_ssl and httpd-tools) and
ran a load test using JMeter.
I have these aggregate numbers for the two machines:
Linux kernel 3.2.38 / httpd 2.2.23
Samples: 9942
Average: 715
Median: 347
90% Line: 1500
Min: 159
Max: 47157
Throughput: 35.1 k/sec
KB/sec: 120.4
Linux 3.2.38 / httpd 2.2.26:
Samples: 9941
Average: 1124
Median: 344
90% line: 2687
Min: 159
Max: 117918
Throughput: 35.1
KB/sec: 120.4
Otherwise the configuration is essentially the same: same modules
loaded, etc. The numbers from above represent essentially a trivial
request to my Tomcat backend.
I'm not sure I can chalk-up the above difference to simply network noise
(specifically, average, max, 90% line). What I'm observing here is
certainly not the extent of the problems we are experiencing, but I have
noticed a difference between these two versions as provided by Amazon's
yum repo.
I'll see what happens if I start to update other components as well.
-chris
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Igor Cicimov <ic...@gmail.com>.
On 06/04/2014 5:51 AM, "Christopher Schultz" <ch...@christopherschultz.net>
wrote:
>
> Igor,
>
> On 4/4/14, 5:39 AM, Igor Cicimov wrote:
> >
> > On 04/04/2014 1:05 AM, "Christopher Schultz"
> > <chris@christopherschultz.net <ma...@christopherschultz.net>>
wrote:
> >>
> >> All,
> >>
> >> I'm having a problem in production I've never seem before. We are
> >> running a pair of AWS EC2 m1.micro web servers where only one of them
in
> >> really in service at any given time. The httpd instance serves some
> >> static content and forwards a great deal of traffic via stunnel to a
> >> single back-end Tomcat server using mod_jk 1.2.37. We have been running
> >> under this configuration for several years with no problems.
> >>
> > Enable the stunnel logs maybe they will reveal something?
>
> I don't think stunnel has changed much. Besides, the stunnel processes
> aren't eating up the CPU: it's the httpd processes that are.
>
Yes but apache actually connects to stunnel and can be affected by it in
case of any issues it has.
I would also try to run pstack again some of the pids consuming the cpu to
find out what are they doing.
Basically since the kernel got upgraded any of the parts used here might be
possibly impacted by some new or existing bug.
> >> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
> >> Amazon's previous version (I can't remember which one), including the
> >> package-refresh that comes with it for httpd. The current kernel
version
> >> is 3.10.34. The current httpd version is 2.2.26. The package name is
> >> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
> >> prefork MPM with the following (default) settings:
> >>
> >> StartServers 8
> >> MinSpareServers 5
> >> MaxSpareServers 20
> >> ServerLimit 256
> >> MaxClients 256
> >> MaxRequestsPerChild 4000
> >>
> >> What I can observe is that the CPU load average is rising from the
usual
> >> sub-2.0 value to sometimes as high as 70. That's seventy, not
> >> seven-point-oh.
> >>
> >> I see no errors in the log, and httpd doesn't seem to be dropping any
> >> requests... just running very very slowly.
> >>
> > What if you increase the LogLevel to debug? Meaybe jkLogLevel too.
>
> I could certainly do that, but the mod_jk binary is the same as before
> the upgrade.
>
> >> It seems to come in waves: the load will go up, and everything will
slow
> >> down, and then we'll get a reprieve.
> >
> > Whats the memory usage at those times? If you have sysstat installed you
> > can run sar for some stats about disk cpu and memory.
>
> I installed sysstat and am collecting data. Monday morning is when we'll
> get killed, and I'll have the data then.
>
> > First thing i would do is move from prefork to mpm worker. You
> > should see significant improvement.
>
> While that may be the case, something else must have changed. We had
> been using the prefork MPM beforehand without a problem.
>
> We've upgraded to Linux kernel 3.10.35 at the suggestion of the AWS
> support folks, but things still look pretty ugly. I've resurrected an
> old snapshot to compare the performance relative to the upgraded
> instance. If the 3.10.35 instance falls-over on Monday, I'll switch-over
> to the older kernel instance.
>
> Thanks,
> -chris
>
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Jeff Trawick <tr...@gmail.com>.
On Sat, Apr 5, 2014 at 3:18 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> Igor,
>
> On 4/4/14, 5:39 AM, Igor Cicimov wrote:
> >
> > On 04/04/2014 1:05 AM, "Christopher Schultz"
> > <chris@christopherschultz.net <ma...@christopherschultz.net>>
> wrote:
> >>
> >> All,
> >>
> >> I'm having a problem in production I've never seem before. We are
> >> running a pair of AWS EC2 m1.micro web servers where only one of them in
> >> really in service at any given time. The httpd instance serves some
> >> static content and forwards a great deal of traffic via stunnel to a
> >> single back-end Tomcat server using mod_jk 1.2.37. We have been running
> >> under this configuration for several years with no problems.
> >>
> > Enable the stunnel logs maybe they will reveal something?
>
> I don't think stunnel has changed much. Besides, the stunnel processes
> aren't eating up the CPU: it's the httpd processes that are.
>
> >> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
> >> Amazon's previous version (I can't remember which one), including the
> >> package-refresh that comes with it for httpd. The current kernel version
> >> is 3.10.34. The current httpd version is 2.2.26. The package name is
> >> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
> >> prefork MPM with the following (default) settings:
> >>
> >> StartServers 8
> >> MinSpareServers 5
> >> MaxSpareServers 20
>
the range of spare servers here is < 10% of your max (256), and if the load
changes by more than 10% httpd will have to kill a child or fork a new one
> >> ServerLimit 256
> >> MaxClients 256
> >> MaxRequestsPerChild 4000
>
4000 is generally considered very low (needless termination/creation of
child processes.
See if you're getting frequent child process creation (server status
report, staring at ps outputs).
If so {
Get rid of it (set MaxSpareServers higher, set MaxRequestsPerChild very
high or disable it by setting it to 0).
Perhaps look at pages 39 and following in
https://blogs.oracle.com/trawick/resource/DeepDive/WebStackDeepDiveApache.pdf?
(Note that there is a StartServers/MinSpareServers blunder in there (should
be MaxSpareServers).)
}
> >>
> >> What I can observe is that the CPU load average is rising from the usual
> >> sub-2.0 value to sometimes as high as 70. That's seventy, not
> >> seven-point-oh.
> >>
> >> I see no errors in the log, and httpd doesn't seem to be dropping any
> >> requests... just running very very slowly.
> >>
> > What if you increase the LogLevel to debug? Meaybe jkLogLevel too.
>
> I could certainly do that, but the mod_jk binary is the same as before
> the upgrade.
>
> >> It seems to come in waves: the load will go up, and everything will slow
> >> down, and then we'll get a reprieve.
> >
> > Whats the memory usage at those times? If you have sysstat installed you
> > can run sar for some stats about disk cpu and memory.
>
> I installed sysstat and am collecting data. Monday morning is when we'll
> get killed, and I'll have the data then.
>
> > First thing i would do is move from prefork to mpm worker. You
> > should see significant improvement.
>
> While that may be the case, something else must have changed. We had
> been using the prefork MPM beforehand without a problem.
>
> We've upgraded to Linux kernel 3.10.35 at the suggestion of the AWS
> support folks, but things still look pretty ugly. I've resurrected an
> old snapshot to compare the performance relative to the upgraded
> instance. If the 3.10.35 instance falls-over on Monday, I'll switch-over
> to the older kernel instance.
>
can you run oprofile or something similar in that environment?
>
> Thanks,
> -chris
>
>
--
Born in Roswell... married an alien...
http://emptyhammock.com/
http://edjective.org/
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Igor,
On 4/4/14, 5:39 AM, Igor Cicimov wrote:
>
> On 04/04/2014 1:05 AM, "Christopher Schultz"
> <chris@christopherschultz.net <ma...@christopherschultz.net>> wrote:
>>
>> All,
>>
>> I'm having a problem in production I've never seem before. We are
>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>> really in service at any given time. The httpd instance serves some
>> static content and forwards a great deal of traffic via stunnel to a
>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>> under this configuration for several years with no problems.
>>
> Enable the stunnel logs maybe they will reveal something?
I don't think stunnel has changed much. Besides, the stunnel processes
aren't eating up the CPU: it's the httpd processes that are.
>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>> Amazon's previous version (I can't remember which one), including the
>> package-refresh that comes with it for httpd. The current kernel version
>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>> prefork MPM with the following (default) settings:
>>
>> StartServers 8
>> MinSpareServers 5
>> MaxSpareServers 20
>> ServerLimit 256
>> MaxClients 256
>> MaxRequestsPerChild 4000
>>
>> What I can observe is that the CPU load average is rising from the usual
>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>> seven-point-oh.
>>
>> I see no errors in the log, and httpd doesn't seem to be dropping any
>> requests... just running very very slowly.
>>
> What if you increase the LogLevel to debug? Meaybe jkLogLevel too.
I could certainly do that, but the mod_jk binary is the same as before
the upgrade.
>> It seems to come in waves: the load will go up, and everything will slow
>> down, and then we'll get a reprieve.
>
> Whats the memory usage at those times? If you have sysstat installed you
> can run sar for some stats about disk cpu and memory.
I installed sysstat and am collecting data. Monday morning is when we'll
get killed, and I'll have the data then.
> First thing i would do is move from prefork to mpm worker. You
> should see significant improvement.
While that may be the case, something else must have changed. We had
been using the prefork MPM beforehand without a problem.
We've upgraded to Linux kernel 3.10.35 at the suggestion of the AWS
support folks, but things still look pretty ugly. I've resurrected an
old snapshot to compare the performance relative to the upgraded
instance. If the 3.10.35 instance falls-over on Monday, I'll switch-over
to the older kernel instance.
Thanks,
-chris
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Igor Cicimov <ic...@gmail.com>.
On 04/04/2014 1:05 AM, "Christopher Schultz" <ch...@christopherschultz.net>
wrote:
>
> All,
>
> I'm having a problem in production I've never seem before. We are
> running a pair of AWS EC2 m1.micro web servers where only one of them in
> really in service at any given time. The httpd instance serves some
> static content and forwards a great deal of traffic via stunnel to a
> single back-end Tomcat server using mod_jk 1.2.37. We have been running
> under this configuration for several years with no problems.
>
Enable the stunnel logs maybe they will reveal something?
> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
> Amazon's previous version (I can't remember which one), including the
> package-refresh that comes with it for httpd. The current kernel version
> is 3.10.34. The current httpd version is 2.2.26. The package name is
> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
> prefork MPM with the following (default) settings:
>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 256
> MaxRequestsPerChild 4000
>
> What I can observe is that the CPU load average is rising from the usual
> sub-2.0 value to sometimes as high as 70. That's seventy, not
> seven-point-oh.
>
> I see no errors in the log, and httpd doesn't seem to be dropping any
> requests... just running very very slowly.
>
What if you increase the LogLevel to debug? Meaybe jkLogLevel too.
> It seems to come in waves: the load will go up, and everything will slow
> down, and then we'll get a reprieve.
>
Whats the memory usage at those times? If you have sysstat installed you
can run sar for some stats about disk cpu and memory.
> I can see 22 server processes running right this moment, but the load
> average has dropped back to 0.05.
>
> I've enabled ExtendedStatus and it really doesn't look like there is a
> huge number of requests being served. Less than 1 req/sec. This is *not*
> a high-load server. I can see some of the httpd child processes using
> 20% or more of the CPU as reported by 'top'.
>
> Is there a good way for me to determine what those processes are doing?
> As this is a modestly-used server, I can probably enable additional
> logging without too much trouble.
>
> Any help anyone can provide would be very much appreciated.
>
> Thanks,
> -chris
>
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Igor Cicimov <ic...@gmail.com>.
On 04/04/2014 1:05 AM, "Christopher Schultz" <ch...@christopherschultz.net>
wrote:
>
> All,
>
> I'm having a problem in production I've never seem before. We are
> running a pair of AWS EC2 m1.micro web servers where only one of them in
> really in service at any given time. The httpd instance serves some
> static content and forwards a great deal of traffic via stunnel to a
> single back-end Tomcat server using mod_jk 1.2.37. We have been running
> under this configuration for several years with no problems.
>
> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
> Amazon's previous version (I can't remember which one), including the
> package-refresh that comes with it for httpd. The current kernel version
> is 3.10.34. The current httpd version is 2.2.26. The package name is
> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
> prefork MPM with the following (default) settings:
>
First thing i would do is move from prefork to mpm worker. You should see
significant improvement.
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 256
> MaxRequestsPerChild 4000
>
> What I can observe is that the CPU load average is rising from the usual
> sub-2.0 value to sometimes as high as 70. That's seventy, not
> seven-point-oh.
>
> I see no errors in the log, and httpd doesn't seem to be dropping any
> requests... just running very very slowly.
>
> It seems to come in waves: the load will go up, and everything will slow
> down, and then we'll get a reprieve.
>
> I can see 22 server processes running right this moment, but the load
> average has dropped back to 0.05.
>
> I've enabled ExtendedStatus and it really doesn't look like there is a
> huge number of requests being served. Less than 1 req/sec. This is *not*
> a high-load server. I can see some of the httpd child processes using
> 20% or more of the CPU as reported by 'top'.
>
> Is there a good way for me to determine what those processes are doing?
> As this is a modestly-used server, I can probably enable additional
> logging without too much trouble.
>
> Any help anyone can provide would be very much appreciated.
>
> Thanks,
> -chris
>
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Jason Cillo <ja...@comcast.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hmmm. Sorry, I think I typed a reply on the wrong post. Someone had posted a question about setting up https from http, that is what I meant to reply to, since I had just that same problem and find this guide useful. I've already deleted that person's message.
...Jason
On Apr 3, 2014, at 11:41 AM, Christopher Schultz wrote:
> Jason,
>
> On 4/3/14, 11:35 AM, Jason Cillo wrote:
>> Have you looked at OpenSSL Cookbook? It's free. I find it very
>> useful. (I even bought the larger book of which it is a chapter.)
>
> Did you have something in particular that you thought would be relevant
> from that book?
>
> -chris
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJTPYMgAAoJECXq3rM/ywR38V0H/iUZqI16Ii0GXiXrb5VcFG1L
F+xjFeHk2JVKwWgQ34uqoQxqEbxoSSqyfXLgqLzX81ar2taM2wLZkXzQxkYM2p6Y
uRN0jrlKH0pm8gV6Etr7WVcKFTbQIzDze2hocAMh0nP6zQiFBljLPshW5MwAHCX4
Y3F4yo6LAzCUz9srFYXeDY/2SsoLx5pMEAXkK66ee5UYkHio1VUhuksr/blpXsot
zDEEYscHw3K73IZ0Q0madfH8NPHpjKpzRXGxs2AG+TLo6dsrK36ZQ+Hg/8z/ECC3
jIJ6dd86Qk0Khhi5aPHP2gjuA+mSxhXpUkzgnApG2p0BDymhQsg5WDI0meM5C4Y=
=ptGf
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jason,
On 4/3/14, 11:35 AM, Jason Cillo wrote:
> Have you looked at OpenSSL Cookbook? It's free. I find it very
> useful. (I even bought the larger book of which it is a chapter.)
Did you have something in particular that you thought would be relevant
from that book?
-chris
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Jason Cillo <ja...@comcast.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Have you looked at OpenSSL Cookbook? It's free. I find it very useful. (I even bought the larger book of which it is a chapter.)
...Jason
On Apr 3, 2014, at 10:23 AM, Christopher Schultz wrote:
> All,
>
> I forgot to mention that most of our traffic is over SSL. OpenSSL
> version is OpenSSL 1.0.1e-fips 11 Feb 2013.
>
> Thanks,
> -chris
>
> On 4/3/14, 10:04 AM, Christopher Schultz wrote:
>> All,
>>
>> I'm having a problem in production I've never seem before. We are
>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>> really in service at any given time. The httpd instance serves some
>> static content and forwards a great deal of traffic via stunnel to a
>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>> under this configuration for several years with no problems.
>>
>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>> Amazon's previous version (I can't remember which one), including the
>> package-refresh that comes with it for httpd. The current kernel version
>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>> prefork MPM with the following (default) settings:
>>
>> StartServers 8
>> MinSpareServers 5
>> MaxSpareServers 20
>> ServerLimit 256
>> MaxClients 256
>> MaxRequestsPerChild 4000
>>
>> What I can observe is that the CPU load average is rising from the usual
>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>> seven-point-oh.
>>
>> I see no errors in the log, and httpd doesn't seem to be dropping any
>> requests... just running very very slowly.
>>
>> It seems to come in waves: the load will go up, and everything will slow
>> down, and then we'll get a reprieve.
>>
>> I can see 22 server processes running right this moment, but the load
>> average has dropped back to 0.05.
>>
>> I've enabled ExtendedStatus and it really doesn't look like there is a
>> huge number of requests being served. Less than 1 req/sec. This is *not*
>> a high-load server. I can see some of the httpd child processes using
>> 20% or more of the CPU as reported by 'top'.
>>
>> Is there a good way for me to determine what those processes are doing?
>> As this is a modestly-used server, I can probably enable additional
>> logging without too much trouble.
>>
>> Any help anyone can provide would be very much appreciated.
>>
>> Thanks,
>> -chris
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
iQEcBAEBAgAGBQJTPX/AAAoJECXq3rM/ywR3K1sH/iKkmuUY4fmpKSgnWA9+ISwF
QEnp4wO7TCVfAILuG9AgHzTftDsSW0Q8KqqTXgJRR3tIiF40yuDDjpG6wK+/L26g
Yi/kWsXZemvJoDHFRX4n3O02YMw4Z+chmSsz+6YNM9uQ6xOObYOxYFCEmHFgRfDH
adg0O4+5LtT3GzqtNflIoXWI42sMPlHi+BXQqrNgWnNBD7OIFew1jbc7CCDXkfhU
ZnDrogv7T0/nJG8cyRH3PdfiQUisQT5wuWEU532Ud0gdN/rvn9UDcjun4VhyEqD0
uVsmrSOH91S5ugLXXBu1QbKqJRl5jbzGrWYHvEhPgXqWwcoRUN399+vG68MvplM=
=lAyz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
All,
This is the current status map on me server:
Total accesses: 14118 - Total Traffic: 3.9 GB
CPU Usage: u404.04 s9.82 cu0 cs0 - .956% CPU load
.326 requests/sec - 94.4 kB/second - 289.5 kB/request
6 requests currently being processed, 15 idle workers
.__..._.._.R_R_......_...__.._._R_.R...R..W_..__................
................................................................
................................................................
................................................................
top reports CPU load average "4.71, 10.49, 10.43" so things are settling
down a bit right now. Here's what top shows for the httpd processes (at
the top of the report, so these are the highest-CPU processes this
particular moment):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23225 apache 20 0 27772 6860 2444 R 8.6 1.1 0:02.94 httpd
23337 apache 20 0 27764 6724 2324 R 6.3 1.1 0:00.80 httpd
23224 apache 20 0 27820 7112 2624 R 4.6 1.2 0:03.02 httpd
23023 apache 20 0 27824 7204 2692 R 4.3 1.2 0:12.15 httpd
21815 apache 20 0 27820 7420 2916 R 4.0 1.2 0:58.10 httpd
22478 apache 20 0 27992 8412 3856 R 4.0 1.4 0:28.18 httpd
I wasn't able to copy/paste all of that stuff at exactly the same time,
so the numbers are a bit off from each other.
But the server is simply not handling very much traffic. I don't
understand the spikes in CPU load -- which is what lows everything down.
Even doing something like attaching strace to a running process took
something like 30 seconds before it started actually logging anything to
the terminal.
I checked the status of entropy availability using this command:
$ cat /proc/sys/kernel/random/entropy_avail
I did this a few tens of times and the lowest number I ever got was 133.
On systems that have truly run out of randomness, I have seen this
number drop to zero (it usually happens during OpenSSL server key
generation :).
I'm at a loss for what to check next. Any suggestions?
Thanks,
-chris
On 4/3/14, 10:38 AM, Christopher Schultz wrote:
> Oscar,
>
> On 4/3/14, 10:33 AM, Oscar Knorn wrote:
>> Sounds like apache is waiting for a response or a means to forward the
>> request via stunnel.
>> Is /dev/random or /dev/urandom providing sufficient random to the process?
>
> I'm not sure how to check that. Any ideas?
>
> In either case above, wouldn't the processes be sitting idle waiting for
> blocking IO to return? I'm not surprised that the bytes are moving
> slowly, actually. I'm surprised that I'm pegging my CPU waiting for
> bytes to move...
>
> Thanks,
> -chris
>
>> On 4/3/14, 40:23 PM, Christopher Schultz wrote:
>>> All,
>>>
>>> I forgot to mention that most of our traffic is over SSL. OpenSSL
>>> version is OpenSSL 1.0.1e-fips 11 Feb 2013.
>>>
>>> Thanks,
>>> -chris
>>>
>>> On 4/3/14, 10:04 AM, Christopher Schultz wrote:
>>>> All,
>>>>
>>>> I'm having a problem in production I've never seem before. We are
>>>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>>>> really in service at any given time. The httpd instance serves some
>>>> static content and forwards a great deal of traffic via stunnel to a
>>>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>>>> under this configuration for several years with no problems.
>>>>
>>>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>>>> Amazon's previous version (I can't remember which one), including the
>>>> package-refresh that comes with it for httpd. The current kernel version
>>>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>>>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>>>> prefork MPM with the following (default) settings:
>>>>
>>>> StartServers 8
>>>> MinSpareServers 5
>>>> MaxSpareServers 20
>>>> ServerLimit 256
>>>> MaxClients 256
>>>> MaxRequestsPerChild 4000
>>>>
>>>> What I can observe is that the CPU load average is rising from the usual
>>>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>>>> seven-point-oh.
>>>>
>>>> I see no errors in the log, and httpd doesn't seem to be dropping any
>>>> requests... just running very very slowly.
>>>>
>>>> It seems to come in waves: the load will go up, and everything will slow
>>>> down, and then we'll get a reprieve.
>>>>
>>>> I can see 22 server processes running right this moment, but the load
>>>> average has dropped back to 0.05.
>>>>
>>>> I've enabled ExtendedStatus and it really doesn't look like there is a
>>>> huge number of requests being served. Less than 1 req/sec. This is *not*
>>>> a high-load server. I can see some of the httpd child processes using
>>>> 20% or more of the CPU as reported by 'top'.
>>>>
>>>> Is there a good way for me to determine what those processes are doing?
>>>> As this is a modestly-used server, I can probably enable additional
>>>> logging without too much trouble.
>>>>
>>>> Any help anyone can provide would be very much appreciated.
>>>>
>>>> Thanks,
>>>> -chris
>>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Oscar,
On 4/3/14, 10:33 AM, Oscar Knorn wrote:
> Sounds like apache is waiting for a response or a means to forward the
> request via stunnel.
> Is /dev/random or /dev/urandom providing sufficient random to the process?
I'm not sure how to check that. Any ideas?
In either case above, wouldn't the processes be sitting idle waiting for
blocking IO to return? I'm not surprised that the bytes are moving
slowly, actually. I'm surprised that I'm pegging my CPU waiting for
bytes to move...
Thanks,
-chris
> On 4/3/14, 40:23 PM, Christopher Schultz wrote:
>> All,
>>
>> I forgot to mention that most of our traffic is over SSL. OpenSSL
>> version is OpenSSL 1.0.1e-fips 11 Feb 2013.
>>
>> Thanks,
>> -chris
>>
>> On 4/3/14, 10:04 AM, Christopher Schultz wrote:
>>> All,
>>>
>>> I'm having a problem in production I've never seem before. We are
>>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>>> really in service at any given time. The httpd instance serves some
>>> static content and forwards a great deal of traffic via stunnel to a
>>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>>> under this configuration for several years with no problems.
>>>
>>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>>> Amazon's previous version (I can't remember which one), including the
>>> package-refresh that comes with it for httpd. The current kernel version
>>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>>> prefork MPM with the following (default) settings:
>>>
>>> StartServers 8
>>> MinSpareServers 5
>>> MaxSpareServers 20
>>> ServerLimit 256
>>> MaxClients 256
>>> MaxRequestsPerChild 4000
>>>
>>> What I can observe is that the CPU load average is rising from the usual
>>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>>> seven-point-oh.
>>>
>>> I see no errors in the log, and httpd doesn't seem to be dropping any
>>> requests... just running very very slowly.
>>>
>>> It seems to come in waves: the load will go up, and everything will slow
>>> down, and then we'll get a reprieve.
>>>
>>> I can see 22 server processes running right this moment, but the load
>>> average has dropped back to 0.05.
>>>
>>> I've enabled ExtendedStatus and it really doesn't look like there is a
>>> huge number of requests being served. Less than 1 req/sec. This is *not*
>>> a high-load server. I can see some of the httpd child processes using
>>> 20% or more of the CPU as reported by 'top'.
>>>
>>> Is there a good way for me to determine what those processes are doing?
>>> As this is a modestly-used server, I can probably enable additional
>>> logging without too much trouble.
>>>
>>> Any help anyone can provide would be very much appreciated.
>>>
>>> Thanks,
>>> -chris
>>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Oscar Knorn <os...@uni-duisburg-essen.de>.
Sounds like apache is waiting for a response or a means to forward the
request via stunnel.
Is /dev/random or /dev/urandom providing sufficient random to the process?
Cheers Oscar
On 4/3/14, 40:23 PM, Christopher Schultz wrote:
> All,
>
> I forgot to mention that most of our traffic is over SSL. OpenSSL
> version is OpenSSL 1.0.1e-fips 11 Feb 2013.
>
> Thanks,
> -chris
>
> On 4/3/14, 10:04 AM, Christopher Schultz wrote:
>> All,
>>
>> I'm having a problem in production I've never seem before. We are
>> running a pair of AWS EC2 m1.micro web servers where only one of them in
>> really in service at any given time. The httpd instance serves some
>> static content and forwards a great deal of traffic via stunnel to a
>> single back-end Tomcat server using mod_jk 1.2.37. We have been running
>> under this configuration for several years with no problems.
>>
>> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
>> Amazon's previous version (I can't remember which one), including the
>> package-refresh that comes with it for httpd. The current kernel version
>> is 3.10.34. The current httpd version is 2.2.26. The package name is
>> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
>> prefork MPM with the following (default) settings:
>>
>> StartServers 8
>> MinSpareServers 5
>> MaxSpareServers 20
>> ServerLimit 256
>> MaxClients 256
>> MaxRequestsPerChild 4000
>>
>> What I can observe is that the CPU load average is rising from the usual
>> sub-2.0 value to sometimes as high as 70. That's seventy, not
>> seven-point-oh.
>>
>> I see no errors in the log, and httpd doesn't seem to be dropping any
>> requests... just running very very slowly.
>>
>> It seems to come in waves: the load will go up, and everything will slow
>> down, and then we'll get a reprieve.
>>
>> I can see 22 server processes running right this moment, but the load
>> average has dropped back to 0.05.
>>
>> I've enabled ExtendedStatus and it really doesn't look like there is a
>> huge number of requests being served. Less than 1 req/sec. This is *not*
>> a high-load server. I can see some of the httpd child processes using
>> 20% or more of the CPU as reported by 'top'.
>>
>> Is there a good way for me to determine what those processes are doing?
>> As this is a modestly-used server, I can probably enable additional
>> logging without too much trouble.
>>
>> Any help anyone can provide would be very much appreciated.
>>
>> Thanks,
>> -chris
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] High CPU Usage in Amazon EC2
Posted by Christopher Schultz <ch...@christopherschultz.net>.
All,
I forgot to mention that most of our traffic is over SSL. OpenSSL
version is OpenSSL 1.0.1e-fips 11 Feb 2013.
Thanks,
-chris
On 4/3/14, 10:04 AM, Christopher Schultz wrote:
> All,
>
> I'm having a problem in production I've never seem before. We are
> running a pair of AWS EC2 m1.micro web servers where only one of them in
> really in service at any given time. The httpd instance serves some
> static content and forwards a great deal of traffic via stunnel to a
> single back-end Tomcat server using mod_jk 1.2.37. We have been running
> under this configuration for several years with no problems.
>
> Last weekend, we upgraded our OS to Amazon Linux 2014.03 (32-bit) from
> Amazon's previous version (I can't remember which one), including the
> package-refresh that comes with it for httpd. The current kernel version
> is 3.10.34. The current httpd version is 2.2.26. The package name is
> "httpd-2.2.26-1.1.amzn1.i686" if anyone is interested. We are using a
> prefork MPM with the following (default) settings:
>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 256
> MaxRequestsPerChild 4000
>
> What I can observe is that the CPU load average is rising from the usual
> sub-2.0 value to sometimes as high as 70. That's seventy, not
> seven-point-oh.
>
> I see no errors in the log, and httpd doesn't seem to be dropping any
> requests... just running very very slowly.
>
> It seems to come in waves: the load will go up, and everything will slow
> down, and then we'll get a reprieve.
>
> I can see 22 server processes running right this moment, but the load
> average has dropped back to 0.05.
>
> I've enabled ExtendedStatus and it really doesn't look like there is a
> huge number of requests being served. Less than 1 req/sec. This is *not*
> a high-load server. I can see some of the httpd child processes using
> 20% or more of the CPU as reported by 'top'.
>
> Is there a good way for me to determine what those processes are doing?
> As this is a modestly-used server, I can probably enable additional
> logging without too much trouble.
>
> Any help anyone can provide would be very much appreciated.
>
> Thanks,
> -chris
>