You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Anders Nielsen <an...@jobindex.dk> on 2005/03/16 10:47:19 UTC
mod_perl server doesn't reply
We are using a setup with a "light-weight" apache in front of the
mod_perl server.
Lately we have seen some problems with the mod_perl server. From time to
time it stops responding to the requests from the front end. The problem
seems to be related to the mod_perl server only since wget-ing on the
localhost:port doesn't work either.
Script with small or no output do work however. The server is also able
to handle 404's and server-status.
To debug I have tried to splitting the mod_perl the url space handled by
mod_perl into 3 parts and creating 3 separate servers to handle these
parts.
The problem occurred again on all 3 servers at the same time. This has
led me to believe that the problem might be related to the operating
system. Perhaps some resource that is being overused. I am out of ideas
however.
Hope this list can help.
Thanks in advance.
Anders Nielsen
Re: mod_perl server doesn't reply
Posted by Anders Nielsen <an...@jobindex.dk>.
On Mon, 2005-03-21 at 16:51 -0500, Stas Bekman wrote:
> > Failed requests: 21
> > (Connect: 0, Length: 21, Exceptions: 0)
>
> Googling helps :) Rich Graves says:
> http://my.brandeis.edu/bboard/q-and-a-fetch-msg?msg_id=0001GK
> (Ignore "failed requests" -- ab considers it a failure when content is
> different, but that should be true for a dynamic page.
>
> So if they were supposed to give the same content, than you have something
> to look at.
>
Ok, no problem then. Actually I did google before mailing but for the
wrong words apparently :-)
> >
> >
> > When testing our development server I sometimes get this error from ab:
> > Completed 1000 requests
> > Completed 2000 requests
> > Completed 3000 requests
> > apr_recv: Connection reset by peer (104)
> > Total of 3320 requests completed
> >
> > I don't know what this mean either. I am not sure why the ab would die
> > of this kind of error instead of just reporting that an error happened.
>
> Could be some problem in Apache or mod_perl. That normally means that the
> server has aborted the connection.
>
Ok, I will look some more at this. It could be related to the problems
we're having. Still I think it's weird that ab quits over this.
Thank you for your time Stas :-)
mvh.
Anders Nielsen
Re: mod_perl server doesn't reply
Posted by Stas Bekman <st...@stason.org>.
Anders Nielsen wrote:
[...]
> I have been doing some testing with ab and I found some interesting
> cases.
>
>
> On our production system I get the following when testing an important
> mod_perl enabled script from my local desktop:
>
> $ ab -n 100 -c 5 "http://www.xxxxxx.dk/cgi/xxxxxx.cgi?q=test&allowall=1"
[...]
> Failed requests: 21
> (Connect: 0, Length: 21, Exceptions: 0)
Googling helps :) Rich Graves says:
http://my.brandeis.edu/bboard/q-and-a-fetch-msg?msg_id=0001GK
(Ignore "failed requests" -- ab considers it a failure when content is
different, but that should be true for a dynamic page.
So if they were supposed to give the same content, than you have something
to look at.
> There are 21 failed requests which is surprising to me. The error type
> is Length. What does this mean? I don't see any explanation of the
> output of ab on the man page.
>
>
>
> When testing our development server I sometimes get this error from ab:
> Completed 1000 requests
> Completed 2000 requests
> Completed 3000 requests
> apr_recv: Connection reset by peer (104)
> Total of 3320 requests completed
>
> I don't know what this mean either. I am not sure why the ab would die
> of this kind of error instead of just reporting that an error happened.
Could be some problem in Apache or mod_perl. That normally means that the
server has aborted the connection.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_perl server doesn't reply
Posted by Anders Nielsen <an...@jobindex.dk>.
On Fri, 2005-03-18 at 17:37 -0500, Stas Bekman wrote:
> Anders, the first priority is to figure out how to reproduce the problem
> at will. If you don't manage that, you most likely are going to waste
> hours staring at the screen. I'd concentrate on that task first.
>
> Try using ab (apachebench), httpd_load or another utility to stress test.
> More advanced tools can record and reply real requests. For some examples see:
> http://modperlbook.org/html/ch09_01.html#pmodperl-CHP-9-SECT-1
>
> Armed with one of these tools you can know exactly how many requests
> you've issued and then making sure that logs/access_log is in agreement.
>
> Also take a look at Apache::Resource which can set certain limits on the
> process resource consumption.
>
I have been doing some testing with ab and I found some interesting
cases.
On our production system I get the following when testing an important
mod_perl enabled script from my local desktop:
$ ab -n 100 -c 5 "http://www.xxxxxx.dk/cgi/xxxxxx.cgi?q=test&allowall=1"
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.121.2.1 $>
apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd,
http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation,
http://www.apache.org/
Benchmarking www.xxxxxx.dk (be patient).....done
Server Software: Apache/1.3.33
Server Hostname: www.xxxxxx.dk
Server Port: 80
Document Path: /cgi/xxxxxx.cgi?q=test&allowall=1
Document Length: 44128 bytes
Concurrency Level: 5
Time taken for tests: 45.82097 seconds
Complete requests: 100
Failed requests: 21
(Connect: 0, Length: 21, Exceptions: 0)
Write errors: 0
Total transferred: 4433142 bytes
HTML transferred: 4412842 bytes
Requests per second: 2.22 [#/sec] (mean)
Time per request: 2254.105 [ms] (mean)
Time per request: 450.821 [ms] (mean, across all concurrent
requests)
Transfer rate: 96.02 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 5 234 766.0 24 3016
Processing: 997 2002 1287.8 1330 6932
Waiting: 36 374 849.1 92 3082
Total: 1015 2237 1529.8 1393 7515
Percentage of the requests served within a certain time (ms)
50% 1393
66% 1756
75% 3162
80% 3670
90% 4298
95% 6101
98% 6948
99% 7515
100% 7515 (longest request)
There are 21 failed requests which is surprising to me. The error type
is Length. What does this mean? I don't see any explanation of the
output of ab on the man page.
When testing our development server I sometimes get this error from ab:
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
apr_recv: Connection reset by peer (104)
Total of 3320 requests completed
I don't know what this mean either. I am not sure why the ab would die
of this kind of error instead of just reporting that an error happened.
Hope you can explain this.
Thanks
Anders
Re: mod_perl server doesn't reply
Posted by Stas Bekman <st...@stason.org>.
Anders, the first priority is to figure out how to reproduce the problem
at will. If you don't manage that, you most likely are going to waste
hours staring at the screen. I'd concentrate on that task first.
Try using ab (apachebench), httpd_load or another utility to stress test.
More advanced tools can record and reply real requests. For some examples see:
http://modperlbook.org/html/ch09_01.html#pmodperl-CHP-9-SECT-1
Armed with one of these tools you can know exactly how many requests
you've issued and then making sure that logs/access_log is in agreement.
Also take a look at Apache::Resource which can set certain limits on the
process resource consumption.
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_perl server doesn't reply
Posted by Anders Nielsen <an...@jobindex.dk>.
On Fri, 2005-03-18 at 11:15 -0500, Stas Bekman wrote:
> Anders, do you by chance use a proxy that may have eaten the response?
>
> What do you see in logs/access_log, it tells how many bytes were sent to
> the client.
>
> What happens if you replace print($x) with Apache->request->print($x)
>
The problem occur when I switch some scripts from mod_cgi to mod_perl.
When I run mod_cgi the front end server takes care of the request. This
setup doesn't have this problem.
I use mod_perl by forwarding (proxying) the request to a back-end
mod_perl server. The mod_perl server is listing on a port on 127.0.0.1.
The problem situation occurs with this setup after a while (usually
after days).
To eliminate the effect of the proxy in my tests I used wget on the same
machine direct on the mod_perl server. I even created a separate
mod_perl server that wasn't used by the front-end. When the problem
situation occurred this independent server also had the problem.
This make me believe that the problem is related to a resource being
depleted. Any ideas on what to monitor?
In the access log I only see lines with the number of bytes being
correct. Example:
127.0.0.1 - - [16/Mar/2005:10:56:20 +0100] "GET /cgise/env.cgi?x=1598223
HTTP/1.0" 200 1598223 "-" "Wget/1.8.2" 0
However it looks like the number of lines in the access log is too low.
This would indicate that some lines are missing. Maybe missing responses
didn't result in a line in the access log. Since I can't reproduce the
problem I can't be sure of this.
If the problem comes back I will look closer at this. I will also
replace the print with Apache->request->print and see what happens.
Thanks again Stas :-)
Re: mod_perl server doesn't reply
Posted by Stas Bekman <st...@stason.org>.
Anders Nielsen wrote:
> On Wed, 2005-03-16 at 21:11 -0500, Stas Bekman wrote:
>
>>Please see:
>>21.7. Hanging Processes: Detection and Diagnostics
>>http://modperlbook.org/html/ch21_07.html
>>
>
>
> Thank you for the reply.
>
> The problem I described happened again and I was able to get some more
> information.
>
> The server isn't actually hanging. I put a print STDERR statement in the
> end of the script and it shows up in the error log. The problems is that
> the client doesn't get the response from the server for some reason.
>
> Our scripst usually ends with a print statement that sends the output to
> STDOUT. When I removed this statement the clients got a (empty) reply.
> Before they where getting no reply at all.
>
> For debugging purposes I created a script with one parameter called $x.
> The script prints the character 'a' as many times as a parameter says
> (print 'a' x $x).
>
> I found out that on small values for $x the script worked while the
> client got no reply for big values. I found the limit (15.982.236) where
> the script worked but if I added 1 the client wouldn't get the reply.
> This limit seemed to change after a while however.
>
> I hope you or somebody else has some idea of what is wrong. Or at least
> can give me something to look for next time the problem occur.
Anders, do you by chance use a proxy that may have eaten the response?
What do you see in logs/access_log, it tells how many bytes were sent to
the client.
What happens if you replace print($x) with Apache->request->print($x)
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_perl server doesn't reply
Posted by Anders Nielsen <an...@jobindex.dk>.
On Wed, 2005-03-16 at 21:11 -0500, Stas Bekman wrote:
> Please see:
> 21.7. Hanging Processes: Detection and Diagnostics
> http://modperlbook.org/html/ch21_07.html
>
Thank you for the reply.
The problem I described happened again and I was able to get some more
information.
The server isn't actually hanging. I put a print STDERR statement in the
end of the script and it shows up in the error log. The problems is that
the client doesn't get the response from the server for some reason.
Our scripst usually ends with a print statement that sends the output to
STDOUT. When I removed this statement the clients got a (empty) reply.
Before they where getting no reply at all.
For debugging purposes I created a script with one parameter called $x.
The script prints the character 'a' as many times as a parameter says
(print 'a' x $x).
I found out that on small values for $x the script worked while the
client got no reply for big values. I found the limit (15.982.236) where
the script worked but if I added 1 the client wouldn't get the reply.
This limit seemed to change after a while however.
I hope you or somebody else has some idea of what is wrong. Or at least
can give me something to look for next time the problem occur.
Thanks
Anders
Re: mod_perl server doesn't reply
Posted by Stas Bekman <st...@stason.org>.
Anders Nielsen wrote:
> We are using a setup with a "light-weight" apache in front of the
> mod_perl server.
>
> Lately we have seen some problems with the mod_perl server. From time to
> time it stops responding to the requests from the front end. The problem
> seems to be related to the mod_perl server only since wget-ing on the
> localhost:port doesn't work either.
>
> Script with small or no output do work however. The server is also able
> to handle 404's and server-status.
>
> To debug I have tried to splitting the mod_perl the url space handled by
> mod_perl into 3 parts and creating 3 separate servers to handle these
> parts.
>
> The problem occurred again on all 3 servers at the same time. This has
> led me to believe that the problem might be related to the operating
> system. Perhaps some resource that is being overused. I am out of ideas
> however.
Please see:
21.7. Hanging Processes: Detection and Diagnostics
http://modperlbook.org/html/ch21_07.html
--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
Re: mod_perl server doesn't reply
Posted by Anders Nielsen <an...@jobindex.dk>.
I would like to add that no messages are seen in the error logs. We are
using Apache/1.3.29 and mod_perl/1.29 on Redhat linux.
mvh
Anders Nielsen