You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by pradeep kumar <pr...@gmail.com> on 2006/08/14 16:47:44 UTC
High CPU utilization on using Apache as a reverse proxy
Hi,
I notice there is a 100% CPU utilization which spikes for sometime and then
goes back to normalcy when Apache is used a reverse proxy. I noticed a mail
thread which seems to be a similar problem
http://mail-archives.apache.org/mod_mbox/httpd-dev/200510.mbox/%3C200510252354.30673.jsnow27@gatesec.net%3E.
One thread of one of the child processes hangs, calling poll() and writev()
system calls so frequently that it causes 100% CPU util. The thread is
trying to writev() to a socked which is in CLOSE_WAIT The child process ends
after time defined by ndd parameter tcp_keepalive_interval and the same
service time can be seen in access_log. The problem disappears then.
Regards,
Pradeep
Re: High CPU utilization on using Apache as a reverse proxy
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
pradeep kumar wrote:
> Hi,
>
> Unfortunately the fix didn't help. The 100% CPU utilization is still
> seen. Can you suggest what else could be causing this large CPU utilization.
Not unless you could follow the directions at
http://httpd.apache.org/dev/debugging.html#backtrace
which may show us where httpd is spinning, leaving the cpu pegged at 100%.
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
Hi,
Unfortunately the fix didn't help. The 100% CPU utilization is still seen.
Can you suggest what else could be causing this large CPU utilization.
Regards,
Pradeep
On 8/21/06, pradeep kumar <pr...@gmail.com> wrote:
>
> Hi Jon,
>
> Thanks for letting me know that this issue wont be seen on HPUX. I'll
> check with the orginal fix that Jeff suggested.
>
> Regards,
> Pradeep
>
>
> On 8/20/06, Jon Snow <js...@gatesec.net> wrote:
> >
> >
> > Pradeep,
> >
> > HP-UX will not be affected by the SSL card issue as it was SUN
> > hardware/driver
> > issue that caused the CPU to spin. I would upgrade to the latest apache
> > version 2.2.3 to eliminate any bugs that may have already been fixed.
> > There
> > were a couple of ssl related ones fixed in 2.0.56 for mod_proxy_http and
> > mod_ssl.
> >
> > Regards,
> > Jon
> >
> > On Saturday 19 August 2006 02:04, pradeep kumar wrote:
> > > Hi Jon,
> > >
> > > Sorry if i mislead you into assuming that I am using Solaris. I am
> > using
> > > HP-UX 11iv2. Is HP-UX also affected? In this case also SSL is being
> > used.
> > >
> > > Regards,
> > > Pradeep
> > >
> > >
> >
> >
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
Hi Jon,
Thanks for letting me know that this issue wont be seen on HPUX. I'll check
with the orginal fix that Jeff suggested.
Regards,
Pradeep
On 8/20/06, Jon Snow <js...@gatesec.net> wrote:
>
>
> Pradeep,
>
> HP-UX will not be affected by the SSL card issue as it was SUN
> hardware/driver
> issue that caused the CPU to spin. I would upgrade to the latest apache
> version 2.2.3 to eliminate any bugs that may have already been fixed.
> There
> were a couple of ssl related ones fixed in 2.0.56 for mod_proxy_http and
> mod_ssl.
>
> Regards,
> Jon
>
> On Saturday 19 August 2006 02:04, pradeep kumar wrote:
> > Hi Jon,
> >
> > Sorry if i mislead you into assuming that I am using Solaris. I am using
> > HP-UX 11iv2. Is HP-UX also affected? In this case also SSL is being
> used.
> >
> > Regards,
> > Pradeep
> >
> >
>
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by Jon Snow <js...@gatesec.net>.
Pradeep,
HP-UX will not be affected by the SSL card issue as it was SUN hardware/driver
issue that caused the CPU to spin. I would upgrade to the latest apache
version 2.2.3 to eliminate any bugs that may have already been fixed. There
were a couple of ssl related ones fixed in 2.0.56 for mod_proxy_http and
mod_ssl.
Regards,
Jon
On Saturday 19 August 2006 02:04, pradeep kumar wrote:
> Hi Jon,
>
> Sorry if i mislead you into assuming that I am using Solaris. I am using
> HP-UX 11iv2. Is HP-UX also affected? In this case also SSL is being used.
>
> Regards,
> Pradeep
>
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
Hi Jon,
Sorry if i mislead you into assuming that I am using Solaris. I am using
HP-UX 11iv2. Is HP-UX also affected? In this case also SSL is being used.
Regards,
Pradeep
On 8/17/06, Jon Snow <js...@gatesec.net> wrote:
>
>
> Pradeep,
>
> I can't remember if I posted the results of the SSL issue and high CPU
> usage.
> We tracked it to a bug in the solaris driver for the Sun daughter card,
> some
> mutex issue if my memory serves me correctly. Sun confirmed and issued a
> patch a while ago. As discussed in the quoted post I did not believe it
> was
> an apache issue for the SSL but I showed dumps for a corresponding write
> issue with a forward proxy in an early anticipation it may be related.
>
> This forward proxy problem I had sounds similar to your issue where apache
> appears not to test a return value for a write but it did not utilise
> large
> CPU. Currently I run 2.2.2 and have observed the same write problem as I
> did
> with 2.0.53. One main issue I have with this is the download continues
> while
> the write fails so effectively the active connection is not in sound
> communication with the passive.
>
> I put the huge memory usage down to the byte range bug as these processes
> were
> nearly always streaming or FTPing large files. I have not seen this memory
> usage with the version I am running even when I see the write problem.
>
> To clarify your issue are you running SSL and if so are you using hardware
> crypto? I assume you are using solaris, is it patched for crypto re: above
> and what version of solaris are you using?
>
> Regards,
> Jon
>
> On Tuesday 15 August 2006 00:47, pradeep kumar wrote:
> > Hi,
> >
> > I notice there is a 100% CPU utilization which spikes for sometime and
> then
> > goes back to normalcy when Apache is used a reverse proxy. I noticed a
> mail
> > thread which seems to be a similar problem
> >
> http://mail-archives.apache.org/mod_mbox/httpd-dev/200510.mbox/%3C200510252
> >354.30673.jsnow27@gatesec.net%3E.
> >
> >
> > One thread of one of the child processes hangs, calling poll() and
> writev()
> > system calls so frequently that it causes 100% CPU util. The thread is
> > trying to writev() to a socked which is in CLOSE_WAIT The child process
> > ends after time defined by ndd parameter tcp_keepalive_interval and the
> > same service time can be seen in access_log. The problem disappears
> then.
> >
> > Regards,
> >
> > Pradeep
>
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by Jon Snow <js...@gatesec.net>.
Pradeep,
I can't remember if I posted the results of the SSL issue and high CPU usage.
We tracked it to a bug in the solaris driver for the Sun daughter card, some
mutex issue if my memory serves me correctly. Sun confirmed and issued a
patch a while ago. As discussed in the quoted post I did not believe it was
an apache issue for the SSL but I showed dumps for a corresponding write
issue with a forward proxy in an early anticipation it may be related.
This forward proxy problem I had sounds similar to your issue where apache
appears not to test a return value for a write but it did not utilise large
CPU. Currently I run 2.2.2 and have observed the same write problem as I did
with 2.0.53. One main issue I have with this is the download continues while
the write fails so effectively the active connection is not in sound
communication with the passive.
I put the huge memory usage down to the byte range bug as these processes were
nearly always streaming or FTPing large files. I have not seen this memory
usage with the version I am running even when I see the write problem.
To clarify your issue are you running SSL and if so are you using hardware
crypto? I assume you are using solaris, is it patched for crypto re: above
and what version of solaris are you using?
Regards,
Jon
On Tuesday 15 August 2006 00:47, pradeep kumar wrote:
> Hi,
>
> I notice there is a 100% CPU utilization which spikes for sometime and then
> goes back to normalcy when Apache is used a reverse proxy. I noticed a mail
> thread which seems to be a similar problem
> http://mail-archives.apache.org/mod_mbox/httpd-dev/200510.mbox/%3C200510252
>354.30673.jsnow27@gatesec.net%3E.
>
>
> One thread of one of the child processes hangs, calling poll() and writev()
> system calls so frequently that it causes 100% CPU util. The thread is
> trying to writev() to a socked which is in CLOSE_WAIT The child process
> ends after time defined by ndd parameter tcp_keepalive_interval and the
> same service time can be seen in access_log. The problem disappears then.
>
> Regards,
>
> Pradeep
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
Hi,
I'll try that and get back with the result.
Thanks,
Pradeep
On 8/16/06, Jeff Trawick <tr...@gmail.com> wrote:
>
> On 8/16/06, pradeep kumar <pr...@gmail.com> wrote:
> >
> > Yep, proxy_connect has been enabled. So we dont have an equivalent patch
> for
> > 2.0? As I mentioned the problem is seen with 2.0.55 and not 2.0.53.
>
> There's no reason the patch can't be put into 2.0... Apply this patch
> and report back on whether or not it resolves your problem, and we can
> see about getting it in the next 2.0.x release. Thanks!
>
> Index: modules/proxy/proxy_connect.c
> ===================================================================
> --- modules/proxy/proxy_connect.c (revision 431472)
> +++ modules/proxy/proxy_connect.c (working copy)
> @@ -339,8 +339,10 @@
> else
> break;
> }
> - else if ((pollevent & APR_POLLERR) || (pollevent &
> APR_POLLHUP))
> + else if ((pollevent & APR_POLLERR) || (pollevent &
> APR_POLLHUP)) {
> + rv = APR_EOF;
> break;
> + }
> }
> else
> break;
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by Jeff Trawick <tr...@gmail.com>.
On 8/16/06, pradeep kumar <pr...@gmail.com> wrote:
>
> Yep, proxy_connect has been enabled. So we dont have an equivalent patch for
> 2.0? As I mentioned the problem is seen with 2.0.55 and not 2.0.53.
There's no reason the patch can't be put into 2.0... Apply this patch
and report back on whether or not it resolves your problem, and we can
see about getting it in the next 2.0.x release. Thanks!
Index: modules/proxy/proxy_connect.c
===================================================================
--- modules/proxy/proxy_connect.c (revision 431472)
+++ modules/proxy/proxy_connect.c (working copy)
@@ -339,8 +339,10 @@
else
break;
}
- else if ((pollevent & APR_POLLERR) || (pollevent & APR_POLLHUP))
+ else if ((pollevent & APR_POLLERR) || (pollevent & APR_POLLHUP)) {
+ rv = APR_EOF;
break;
+ }
}
else
break;
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
Hi,
Yep, proxy_connect has been enabled. So we dont have an equivalent patch for
2.0? As I mentioned the problem is seen with 2.0.55 and not 2.0.53.
Thanks and Regards,
Pradeep
On 8/14/06, Jeff Trawick <tr...@gmail.com> wrote:
>
> On 8/14/06, pradeep kumar <pr...@gmail.com> wrote:
> >
> > More info..
> >
> > The problem is seen with 2.0.55. It wasn't seen with 2.0.53. Haven't
> tried
> > 2.0.58 or 2.0.59 yet. The change log for 2.0.58 doesn't seem to indicate
> any
> > fix for this problem.
>
> Here is a patch for the proxy connect path for a similar problem:
>
>
> http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_connect.c?r1=154199&r2=306878&sortby=date
>
> The fix is not in 2.0.x. Do you have proxy connect enabled?
>
Re: High CPU utilization on using Apache as a reverse proxy
Posted by Jeff Trawick <tr...@gmail.com>.
On 8/14/06, pradeep kumar <pr...@gmail.com> wrote:
>
> More info..
>
> The problem is seen with 2.0.55. It wasn't seen with 2.0.53. Haven't tried
> 2.0.58 or 2.0.59 yet. The change log for 2.0.58 doesn't seem to indicate any
> fix for this problem.
Here is a patch for the proxy connect path for a similar problem:
http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/proxy/mod_proxy_connect.c?r1=154199&r2=306878&sortby=date
The fix is not in 2.0.x. Do you have proxy connect enabled?
Re: High CPU utilization on using Apache as a reverse proxy
Posted by pradeep kumar <pr...@gmail.com>.
More info..
The problem is seen with 2.0.55. It wasn't seen with 2.0.53. Haven't tried
2.0.58 or 2.0.59 yet. The change log for 2.0.58 doesn't seem to indicate any
fix for this problem.
On 8/14/06, pradeep kumar <pr...@gmail.com> wrote:
>
> Hi,
>
> I notice there is a 100% CPU utilization which spikes for sometime and
> then goes back to normalcy when Apache is used a reverse proxy. I noticed a
> mail thread which seems to be a similar problem
> http://mail-archives.apache.org/mod_mbox/httpd-dev/200510.mbox/%3C200510252354.30673.jsnow27@gatesec.net%3E.
>
>
> One thread of one of the child processes hangs, calling poll()
> and writev() system calls so frequently that it causes 100% CPU util. The
> thread is trying to writev() to a socked which is in CLOSE_WAIT The child
> process ends after time defined by ndd parameter tcp_keepalive_interval and
> the same service time can be seen in access_log. The problem disappears
> then.
>
> Regards,
>
> Pradeep
>