You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2006/01/23 23:51:49 UTC
DO NOT REPLY [Bug 38359] New: - httpd looping
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
Summary: httpd looping
Product: Apache httpd-2
Version: 2.0.55
Platform: Other
URL: http://suze.ucs.louisiana.edu:8080/apache-loop-bug/
OS/Version: Solaris
Status: NEW
Severity: major
Priority: P2
Component: Core
AssignedTo: bugs@httpd.apache.org
ReportedBy: apachebugs@louisiana.edu
I'm running 2.0.55 with the worker MPM under Solaris 8. I have a large PHP
application, Moodle 1.5.3+, that uses PHP 4.4.1 and Turck-mmcache-2.4.6.
Occasionally, I find that one or more cpus are devoted 100% to an httpd task,
that is, the httpd daemon seems to be looping. There are no error-log entries I
can find to explain this. The Moodle application seems to continue running OK.
I can kill -9 the looping process(es), and this seems to solve the symptom (of
cpu load). Since the server is a SunFire 890, I have 4 dual-core cpus. One cpu
running flat-out is 12.5% of system cpu resources.
The URL above is a directory containing various traces: a script with prstat and
truss output, a core from the gcore command, and a script of some dbx commands
and their output. It seems to be the loop1 example is a tight loop that calls
semop() in libc and gets an EACCESS error. Same for the second looping
instance. But exactly what is repeatedly calling semop() isn't clear to me.
I also put both the Apache and PHP configuration scripts I used, and their
output, in the above-mentioned directory.
I use this same Apache build on other servers here on campus, and have not had
any problems ... I think the load present on the Moodle server is key to this
behaviour.
One other factor seems related: some time back, I was occasionally getting
runaway httpd procs and also these error msgs in the error-log:
Deadlock situation detected/avoided: Failed to acquire global mutex
Deadlock situation detected/avoided: apr_proc_mutex_lock failed.
I found the commentary for Bugzilla Bug 32325, and added this to httpd.conf:
AcceptMutex pthread
This greatly reduced the instances of runaways, but not completely, hence this
report. I seem to be able to reproduce this problem on most busy days, and
I am willing to insert debugging code, or try other Mutex methods ...
HELP!
-- James Dugal
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
------- Additional Comments From apachebugs@louisiana.edu 2006-01-26 17:29 -------
Thanks very much for your help! I now understand much more about tracking
threads ... an old dog has learned a new trick!
As to PHP, this actually seems to be in Turck-MMcache, and I begin to think that
either I have configured it badly, or it has the flaw, not PHP per-se. Ironic
that turch-mmcache is one of those external routines loaded by PHP...
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
------- Additional Comments From rpluem@apache.org 2006-01-24 22:35 -------
What we actually need is a backtrace of the looping thread. Please see:
http://httpd.apache.org/dev/debugging.html
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
wrowe@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |NEEDINFO
------- Additional Comments From wrowe@apache.org 2006-01-24 19:15 -------
I see no backtraces attached to this incident.
Note you will need to provide backtraces on all the threads of the hung process,
so it might be wise to drop the threads per process to some reasonable size such
as 5.
Finally, folks are often confused that it's the mutex code causing the problem.
If you look at most threads they SHOULD be sitting at a mutex, that means they
have no work to do at that instant, and are therefore chewing no CPU.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
------- Additional Comments From apachebugs@louisiana.edu 2006-01-25 02:04 -------
Created an attachment (id=17495)
--> (http://issues.apache.org/bugzilla/attachment.cgi?id=17495&action=view)
prstat, truss, and prstat output from Solaris 2.8
Aha! That debugging page was most helpful! I hope the trace info attached is
useful in knowing what to do next!
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
------- Additional Comments From apachebugs@louisiana.edu 2006-01-24 20:38 -------
I put lots of truss and dbx output, as well as the core files from a gcore, inthe
site given by the URL spec at the front of the original report (ie, it is
http://suze.ucs.louisiana.edu:8080/apache-loop-bug/ )
Is this acceptable? If not I could attach the more interesting ones...
The 3 instances I document involved 4 threads remaining in a looping dhcpd
process. THe loop may be within libthread.so, invoking semop() repeatedly.
I used dbx on the gcore and displayed the threads and lwps to try to provide
context.
I'll gladly provide more info if you tell me what you need!
--James
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |RESOLVED
Resolution| |INVALID
------- Additional Comments From rpluem@apache.org 2006-01-25 23:09 -------
Thanks. Yes this helps, but you may not like the result. So do not shoot the
messenger :-).
The problem seems to be lwp# 11, as seen from truss:
21826/11: 809.8577 semop(983041, 0xFCD00AA8, 1) Err#13 EACCES
As you can see from the pstack output lwp# 11 is deep inside of php code:
----------------- lwp# 11 / thread# 9 --------------------
fed9c054 _semsys (fa800018, 1, 0, 0, 0, 0) + 8
feb92428 _mm_lock (fa800000, 1, 19a758, 0, 0, 0) + 8
feb742ec ???????? (fcd00e78, fcd01278, a4, 43d69f17, fcd00e78, fcd00d24)
feb798c4 ???????? (fcd00e78, fcd01278, fcd00d24, 43d6980f, 3cd398, 0)
feb7a0d8 mmcache_compile_file (fcd01904, 2, 3cd398, 41b600, 41b618, 3c0910) + 104
fd95a7a0 zend_execute_scripts (8, 3cd398, 0, 3, 18, a6b500) + 80
fd90a464 php_execute_script (41b624, 3d2ff0, 166da0, 0, fda1deb8, 9) + 3ec
fd9776fc ???????? (4, fd9fcf84, 0, c6e0b0, 41b600, fda1e0c8)
0002fdfc ap_run_handler (c6e0b0, 3b, c6fb48, c6f2f0, 0, 3b) + 40
00030454 ap_invoke_handler (c6e0b0, 0, c6e0b0, 97df8, 0, 0) + f0
0002b5a8 ap_process_request (c6e0b0, 4, 4, c6e0b0, c8, 97ce8) + 54
00026224 ap_process_http_connection (3d9ae0, 14fc38, 3d99f0, c4, 3d9ad8,
3db9c0) + 60
0003b018 ap_run_process_connection (3d9ae0, 3d99f0, 3d99f0, c4, 3d9ad8, 3db9c0)
+ 40
0002c79c process_socket (3d99b8, 3d99f0, 3, 4, 3db9c0, 73c00) + 60
0002ced8 worker_thread (12f6b8, 0, 0, 72c1c, 10, fe401000) + 174
fee0b01c _thread_start (12f6b8, 0, 0, 0, 0, 0) + 40
So Nick was correct at least with his assumption that it is a php problem.
Please report this bug to http://bugs.php.net/.
Anyway thank you for using Apache httpd server.
One last hint: Try to use ipcs to find out more about the semaphore. The first
parameter from the truss output of the failing semop is the semid that gets
displayed by ipcs:
21826/11: 809.8577 semop(983041, 0xFCD00AA8, 1) Err#13 EACCES
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
apachebugs@louisiana.edu changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|INVALID |
------- Additional Comments From apachebugs@louisiana.edu 2006-01-24 18:58 -------
Are you spouting the "accepted wisdom" or did you bother to examine the traces I
provided? As far as I know, PHP if properly compiled, IS thread-safe, and the worry
is that some external library would NOT be thread safe. All the external ones
I linked with I compiled myself with attention to making sure they were thread
safe (eg, mysql and mcrypt).
I am trying to see where the loop is invoked from the mutex code in Apache,
the suspicion being that it is not properly recovering from an error. Do you
understand what I am suggesting?
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38359] - httpd looping
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38359>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38359
nick@webthing.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |INVALID
Summary|httpd looping |httpd looping
------- Additional Comments From nick@webthing.com 2006-01-24 05:34 -------
Running PHP in a threaded MPM is known to be unsafe. Please use Prefork or
limit yourself to thread-safe applications.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org