general/8908: Apache seems to stop responding after a couples of hours of load test

[Vinay Aggarwal <vi...@hotmail.com>]

>Number:         8908
>Category:       general
>Synopsis:       Apache seems to stop responding after a couples of hours of load test
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    apache
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   apache
>Arrival-Date:   Fri Nov 30 01:20:00 PST 2001
>Closed-Date:
>Last-Modified:
>Originator:     vinayagg@hotmail.com
>Release:        1.3.22
>Organization:
apache
>Environment:
Apache 1.3.22 precompiled binaries for Win2000 SP2.

c:\>uname -a
CYGWIN_NT-5.0 VINAY 1.3.5(0.47/3/2) 2001-11-13 23:16 i686 unknown
>Description:
I have Apache 1.3.22 linked to Tomcat3.3 using mod_jk 
I have been doing a load test on tomcat which essentially 
puts apache and mod_jk also under load test. However I saw that
after a couple of hours Apache stops responding. Here is an extract
from error.log

[Fri Nov 30 01:23:12 2001] [error] [client 65.84.246.20] File does not exist: c:/software/apache/htdocs/scripts/..%5c/winnt/system32/cmd.exe
[Fri Nov 30 01:23:12 2001] [error] [client 65.84.246.20] File does not exist: c:/software/apache/htdocs/scripts/..%2f/winnt/system32/cmd.exe
[Fri Nov 30 01:33:17 2001] [error] forcing termination of child #0 (handle 548)
[Fri Nov 30 01:33:55 2001] [error] [client 127.0.0.1] File does not exist: c:/software/apache/htdocs/robots.txt
[Fri Nov 30 01:57:12 2001] [error] [client 127.0.0.1] File does not exist: c:/software/apache/htdocs/robots.txt
[Fri Nov 30 02:36:32 2001] [error] forcing termination of child #0 (handle 548)

Here is the corresponding access.log for same time frame

65.13.149.233 - - [30/Nov/2001:02:32:44 -0600] "GET /humdum/style.css HTTP/1.1" 200 2842
65.13.149.233 - - [30/Nov/2001:02:32:45 -0600] "POST /humdum/MyBookmarksControl.jsp HTTP/1.1" 200 5580
65.13.149.233 - - [30/Nov/2001:02:32:45 -0600] "GET /humdum/SuggestedProfiles.jsp HTTP/1.1" 200 5614
65.13.149.233 - - [30/Nov/2001:02:32:45 -0600] "POST /humdum/EditPhotoControl.jsp HTTP/1.1" 200 5581
65.13.149.233 - - [30/Nov/2001:02:36:38 -0600] "POST /humdum/SearchProfileAdvControl.jsp HTTP/1.1" 200 4011
65.13.149.233 - - [30/Nov/2001:02:36:39 -0600] "GET /humdum/EditPartnerProfile.jsp HTTP/1.1" 200 5586
65.13.149.233 - - [30/Nov/2001:02:36:39 -0600] "POST /humdum/EditPartnerProfileControl.jsp HTTP/1.1" 200 5586
65.13.149.233 - - [30/Nov/2001:02:36:39 -0600] "GET /humdum/images/v1lKwUBd1006911753328.jpg HTTP/1.1" 200 10579
65.13.149.233 - - [30/Nov/2001:02:36:39 -0600] "GET /humdum/style.css HTTP/1.1" 200 2842
65.13.149.233 - - [30/Nov/2001:02:36:39 -0600] "GET /humdum/EditPartnerProfile.jsp HTTP/1.1" 200 5586

If you closely watch it, you can see that Apache didnt respond for 4 minutes. 
In fact this crash I only figured after I saw at these log files. I thought
it never recovers once it crashes. However yesterday, after crash, I have
seen two processes with name "apache.exe" in my process list. Not sure
how did two processes got invoked. Actually even right now there are two 
processes with same name. I think Apache might be invoking another process
as a failover once it crashes (sorry dont know the architecture of Apache
at all).

My load test involves 200 users hitting the pages with random delays.
My system processor load is about 15% on average. Memory utilization is 
low at 232MB (I have 256MB physical RAM + virtual RAM)
>How-To-Repeat:
Link Apache with Tomcat3.3 using mod_jk, put the system under load.
Hopefully it will repeat itself.
>Fix:
no
>Release-Note:
>Audit-Trail:
>Unformatted:
 [In order for any reply to be added to the PR database, you need]
 [to include <ap...@Apache.Org> in the Cc line and make sure the]
 [subject line starts with the report component and number, with ]
 [or without any 'Re:' prefixes (such as "general/1098:" or      ]
 ["Re: general/1098:").  If the subject doesn't match this       ]
 [pattern, your message will be misfiled and ignored.  The       ]
 ["apbugs" address is not added to the Cc line of messages from  ]
 [the database automatically because of the potential for mail   ]
 [loops.  If you do not include this Cc, your reply may be ig-   ]
 [nored unless you are responding to an explicit request from a  ]
 [developer.  Reply only with text; DO NOT SEND ATTACHMENTS!     ]