mod_rewrite/7105: very sporadically httpd processes peg the cpu

[Cott Lang <co...@internetstaff.com>]

>Number:         7105
>Category:       mod_rewrite
>Synopsis:       very sporadically httpd processes peg the cpu
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    apache
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   apache
>Arrival-Date:   Fri Jan 19 19:50:00 PST 2001
>Closed-Date:
>Last-Modified:
>Originator:     cott@internetstaff.com
>Release:        1.3.6-1.3.12
>Organization:
apache
>Environment:
Solaris 7 x86, Sun cc compiler.
>Description:
Every few days, 1 or more webservers goes nuts with 1 or more processes consuming an entire processor. stack dumps (via pstack) make this appear to be a recursive regex call that goes nuts (although this is the COMPLETE stack dump - so it doesn't appear to recurse many times):

 dfb31ae5 match_re_C () + 208d
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb31523 match_re_C () + 1acb
 dfb31b02 match_re_C () + 20aa
 dfb30274 match_re_C () + 81c
 dfb2f992 __regexec_C (dfb53c2c, 819ad98, 80412ac, 2, 8041134, 0) + d2
 dfb332de regexec  () + 2e
 080ad553 ap_regexec (819ad98, 80412ac, 2, 8041134, 0) + 17
 08094c57 ???????? ()
 080940d7 ???????? ()
 08093c00 ???????? ()
 0809302b ???????? (81d59e8)
 0809cc2b ???????? ()
 0809cc7b ap_translate_name (81d59e8) + 13
 080ab400 ap_sub_req_method_uri () + dc
 080ab50b ap_sub_req_lookup_uri () + 13
 08091616 ???????? ()
 0809cf2a ap_invoke_handler () + 92
 080abd73 ???????? ()
 080abece ap_process_request (81d39c8) + 2a
 080a5554 ???????? (a, f, 80a41d4, 10, 80a41d4, 1)
 080a56e0 ???????? ()
 080a5ccf ???????? ()
 080a62b5 ???????? (1, 8047c10, 816aae8, 816ab10)
 080a687e main     (1, 8047c10, 8047c18) + 346
 08073c4c ???????? ()

I use a few relatively simple rewrite rules that activate on almost all URLs on a site that gets millions of hits a day. This has been going on for a year and a half, but sporadically enough that we've never pursued it. Sometimes it disappears for months, then occurs every day for weeks.

At one time, I dumped core into gdb and debugged quite a few processes on different machines to check the URL getting passed into regexec - they were all VERY normal image or html calls that we process hundreds of thousands of times a day. I'm lost. I can't find any references to this problem anywhere.
>How-To-Repeat:
I have no idea - but it repeats for me regularly. I'm willing to do the legwork to help track it down, but I'm lost at this point.
>Fix:
nope.
>Release-Note:
>Audit-Trail:
>Unformatted:
 [In order for any reply to be added to the PR database, you need]
 [to include <ap...@Apache.Org> in the Cc line and make sure the]
 [subject line starts with the report component and number, with ]
 [or without any 'Re:' prefixes (such as "general/1098:" or      ]
 ["Re: general/1098:").  If the subject doesn't match this       ]
 [pattern, your message will be misfiled and ignored.  The       ]
 ["apbugs" address is not added to the Cc line of messages from  ]
 [the database automatically because of the potential for mail   ]
 [loops.  If you do not include this Cc, your reply may be ig-   ]
 [nored unless you are responding to an explicit request from a  ]
 [developer.  Reply only with text; DO NOT SEND ATTACHMENTS!     ]