You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Mike Jakubik <mi...@intertainservices.com> on 2011/05/17 23:33:00 UTC
apr 1.4.4 breaks mod_jk
Hello,
I have recently discovered while updating my apache22 installation and
subsequently apr1 from 1.4.2 to 1.4.4 that mod_jk now causes the apache
server to consume 100% cpu and become unresponsive. I have tried
recompiling mod_jk with the new apr but that did not help. Below is my
environment information.
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
11241 www 1 114 0 61492K 11200K CPU0 0 0:53 78.96%
httpd
11237 www 1 110 0 61492K 11276K RUN 1 0:40 66.26%
httpd
11238 www 1 108 0 61492K 11276K RUN 0 0:50 56.05%
httpd
Process takes up all resources and is unresponsive, no errors are
logged. Removing mod_jk from apache solves the problem.
FreeBSD staging.local 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Mar 3
17:32:06 EST 2011 root@staging.local:/usr/obj/usr/src/sys/STAGING
amd64
Apache/2.2.18 (FreeBSD) mod_jk/1.2.31 PHP/5.3.6 with Suhosin-Patch
mod_ssl/2.2.18 OpenSSL/0.9.8q
Thanks.
RE: apr 1.4.4 breaks mod_jk
Posted by "Anthony J. Biacco" <ab...@formatdynamics.com>.
Just fyi, I have had no problems yet on apache 2.2.18 worker and apr
1.4.4 (centos 5.6 x86_64 and jk 1.2.31)
-Tony
---------------------------
Manager, IT Operations
Format Dynamics, Inc.
P: 303-228-7327
F: 303-228-7305
abiacco@formatdynamics.com
http://www.formatdynamics.com
> -----Original Message-----
> From: Rainer Jung [mailto:rainer.jung@kippdata.de]
> Sent: Tuesday, May 17, 2011 4:08 PM
> To: dev@apr.apache.org
> Subject: Re: apr 1.4.4 breaks mod_jk
>
> On 17.05.2011 23:33, Mike Jakubik wrote:
> > Hello,
> >
> > I have recently discovered while updating my apache22 installation
and
> > subsequently apr1 from 1.4.2 to 1.4.4 that mod_jk now causes the
apache
> > server to consume 100% cpu and become unresponsive. I have tried
> > recompiling mod_jk with the new apr but that did not help. Below is
my
> > environment information.
> >
> > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
> > COMMAND
> > 11241 www 1 114 0 61492K 11200K CPU0 0 0:53 78.96%
> > httpd
> > 11237 www 1 110 0 61492K 11276K RUN 1 0:40 66.26%
> > httpd
> > 11238 www 1 108 0 61492K 11276K RUN 0 0:50 56.05%
> > httpd
> >
> > Process takes up all resources and is unresponsive, no errors are
> > logged. Removing mod_jk from apache solves the problem.
> >
> > FreeBSD staging.local 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Mar 3
> > 17:32:06 EST 2011
root@staging.local:/usr/obj/usr/src/sys/STAGING
> > amd64
> > Apache/2.2.18 (FreeBSD) mod_jk/1.2.31 PHP/5.3.6 with Suhosin-Patch
> > mod_ssl/2.2.18 OpenSSL/0.9.8q
>
> Thanks for the report. Would you mind opening an issues in Bugzilla?
You
> can choose the Tomcat project and Tomcat Connectors as a component. If
> it turns out as an APR bug, we can move the issue there later.
>
> It would be very helpful, if you could provide thread stacks using
> pstack or gdb. Please provide a couple of such threads stack outputs.
>
> Furthermore, if your MPM is worker, not prefork, it would be helpful
if
> you could check using ps, which of the threads in the process actually
> consume the CPU. I'm not sure about FreeBSD ps flags, but -H -p ...
> could be right. This helps us select out the right thread stack.
>
> Regards,
>
> Rainer
>
Re: apr 1.4.4 breaks mod_jk
Posted by Mike Jakubik <mi...@intertainservices.com>.
On Wed, 2011-05-18 at 00:53 +0200, Rainer Jung wrote:
> If you can find pstack, that will be easier. pstack just takes the
> process id and writes out the stacks for all threads. My quick search
> indicates it pstack should exist for FreeBSD.
>
There is a port for this but it only support i386 arch, im running
amd64.
http://www.freshports.org/sysutils/pstack/
> GDB allows to also get details on the values of variables used, step
> through the code etc. You give gdb two arguments, the path to your httpd
> binary and the process ID of the CPU consuming process. Once the gdb
> prompt is shown, you can enter
>
> thread apply all bt full
>
> which will output thread and variable information for all threads.
>
> pstack and gdb might produce many lines of output, especially for
> worker, so you might want to write the result directly to a file, e.g.
>
> pstack PID > /tmp/pstack.txt
>
> And yes, if the problem also occurs for prefork, then that makes
> debugging a bit easier.
I'll recompile to prefork and give GDB a go, thanks for the info.
Re: apr 1.4.4 breaks mod_jk
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, May 18, 2011 at 2:57 PM, Mike Jakubik
<mi...@intertainservices.com> wrote:
> On Wed, 2011-05-18 at 00:53 +0200, Rainer Jung wrote:
>
> GDB allows to also get details on the values of variables used, step
> through the code etc. You give gdb two arguments, the path to your httpd
> binary and the process ID of the CPU consuming process. Once the gdb
> prompt is shown, you can enter
>
> thread apply all bt full
>
>
> I've recompiled apache and mod_jk with apr 1.4.4, also removed php and
> switched to prefork to make debugging easier, below is the GDB output of the
> process that is hanging at 100% cpu.
>
> Apache starts ok and does not start eating at the cpu untill an http request
> is made to it.
>
>
> (gdb) thread apply all bt full
>
> Thread 1 (Thread 8014041c0 (LWP 100847/httpd)):
> #0 0x0000000801157c1e in strchr () from /lib/libc.so.7
> No symbol table info available.
> #1 0x0000000800d2430f in apr_fnmatch (pattern=0x80155863a "/WEB-INF/*",
> string=0x803745c16 "", flags=Variable "flags" is not available.
> ) at strings/apr_fnmatch.c:230
> escape = 1
> slash = Variable "slash" is not available.
>
> Please let me know if i should run any other commands in GDB.
You're covered by a known regression. Stay subscribed; you'll hear
about a fix soon.
Re: apr 1.4.4 breaks mod_jk
Posted by Mike Jakubik <mi...@intertainservices.com>.
On Wed, 2011-05-18 at 00:53 +0200, Rainer Jung wrote:
> GDB allows to also get details on the values of variables used, step
> through the code etc. You give gdb two arguments, the path to your httpd
> binary and the process ID of the CPU consuming process. Once the gdb
> prompt is shown, you can enter
>
> thread apply all bt full
>
I've recompiled apache and mod_jk with apr 1.4.4, also removed php and
switched to prefork to make debugging easier, below is the GDB output of
the process that is hanging at 100% cpu.
Apache starts ok and does not start eating at the cpu untill an http
request is made to it.
(gdb) thread apply all bt full
Thread 1 (Thread 8014041c0 (LWP 100847/httpd)):
#0 0x0000000801157c1e in strchr () from /lib/libc.so.7
No symbol table info available.
#1 0x0000000800d2430f in apr_fnmatch (pattern=0x80155863a "/WEB-INF/*",
string=0x803745c16 "", flags=Variable "flags" is not available.
) at strings/apr_fnmatch.c:230
escape = 1
slash = Variable "slash" is not available.
Please let me know if i should run any other commands in GDB.
Thanks.
Re: apr 1.4.4 breaks mod_jk
Posted by Rainer Jung <ra...@kippdata.de>.
On 18.05.2011 00:30, Mike Jakubik wrote:
> On Wed, 2011-05-18 at 00:07 +0200, Rainer Jung wrote:
>
>> Thanks for the report. Would you mind opening an issues in Bugzilla? You
>> can choose the Tomcat project and Tomcat Connectors as a component. If
>> it turns out as an APR bug, we can move the issue there later.
>>
>> It would be very helpful, if you could provide thread stacks using
>> pstack or gdb. Please provide a couple of such threads stack outputs.
>>
>> Furthermore, if your MPM is worker, not prefork, it would be helpful if
>> you could check using ps, which of the threads in the process actually
>> consume the CPU. I'm not sure about FreeBSD ps flags, but -H -p ...
>> could be right. This helps us select out the right thread stack.
>>
>> Regards,
>>
>> Rainer
>>
>>
>
>
> I can recompile apache to prefork if it will make debugging easier (fyi,
> problem occurs with both worker and prefork MPMs) . Some help with the
> actual debugging would be appreciated, i can use GDB but i don’t know
> how to preform the debugging process.
If you can find pstack, that will be easier. pstack just takes the
process id and writes out the stacks for all threads. My quick search
indicates it pstack should exist for FreeBSD.
GDB allows to also get details on the values of variables used, step
through the code etc. You give gdb two arguments, the path to your httpd
binary and the process ID of the CPU consuming process. Once the gdb
prompt is shown, you can enter
thread apply all bt full
which will output thread and variable information for all threads.
pstack and gdb might produce many lines of output, especially for
worker, so you might want to write the result directly to a file, e.g.
pstack PID > /tmp/pstack.txt
And yes, if the problem also occurs for prefork, then that makes
debugging a bit easier.
Regards,
Rainer
Re: apr 1.4.4 breaks mod_jk
Posted by Mike Jakubik <mi...@intertainservices.com>.
On Wed, 2011-05-18 at 00:07 +0200, Rainer Jung wrote:
> Thanks for the report. Would you mind opening an issues in Bugzilla? You
> can choose the Tomcat project and Tomcat Connectors as a component. If
> it turns out as an APR bug, we can move the issue there later.
>
> It would be very helpful, if you could provide thread stacks using
> pstack or gdb. Please provide a couple of such threads stack outputs.
>
> Furthermore, if your MPM is worker, not prefork, it would be helpful if
> you could check using ps, which of the threads in the process actually
> consume the CPU. I'm not sure about FreeBSD ps flags, but -H -p ...
> could be right. This helps us select out the right thread stack.
>
> Regards,
>
> Rainer
>
>
I can recompile apache to prefork if it will make debugging easier (fyi,
problem occurs with both worker and prefork MPMs) . Some help with the
actual debugging would be appreciated, i can use GDB but i don’t know
how to preform the debugging process.
Thanks.
Re: apr 1.4.4 breaks mod_jk
Posted by Rainer Jung <ra...@kippdata.de>.
On 17.05.2011 23:33, Mike Jakubik wrote:
> Hello,
>
> I have recently discovered while updating my apache22 installation and
> subsequently apr1 from 1.4.2 to 1.4.4 that mod_jk now causes the apache
> server to consume 100% cpu and become unresponsive. I have tried
> recompiling mod_jk with the new apr but that did not help. Below is my
> environment information.
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> 11241 www 1 114 0 61492K 11200K CPU0 0 0:53 78.96%
> httpd
> 11237 www 1 110 0 61492K 11276K RUN 1 0:40 66.26%
> httpd
> 11238 www 1 108 0 61492K 11276K RUN 0 0:50 56.05%
> httpd
>
> Process takes up all resources and is unresponsive, no errors are
> logged. Removing mod_jk from apache solves the problem.
>
> FreeBSD staging.local 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Mar 3
> 17:32:06 EST 2011 root@staging.local:/usr/obj/usr/src/sys/STAGING
> amd64
> Apache/2.2.18 (FreeBSD) mod_jk/1.2.31 PHP/5.3.6 with Suhosin-Patch
> mod_ssl/2.2.18 OpenSSL/0.9.8q
Thanks for the report. Would you mind opening an issues in Bugzilla? You
can choose the Tomcat project and Tomcat Connectors as a component. If
it turns out as an APR bug, we can move the issue there later.
It would be very helpful, if you could provide thread stacks using
pstack or gdb. Please provide a couple of such threads stack outputs.
Furthermore, if your MPM is worker, not prefork, it would be helpful if
you could check using ps, which of the threads in the process actually
consume the CPU. I'm not sure about FreeBSD ps flags, but -H -p ...
could be right. This helps us select out the right thread stack.
Regards,
Rainer
Re: apr 1.4.4 breaks mod_jk
Posted by Mike Jakubik <mi...@intertainservices.com>.
On Tue, 2011-05-17 at 17:52 -0400, Jeff Trawick wrote:
>
> you updated Apache 2.2.x from 2.2.what to 2.2.what?
>
2.2.17 to 2.2.18. I can however also reproduce the problem in 2.2.17, it
seems to be specific to the apr version.
> (you updated apr 1.4.2 to 1.4.4)
>
> >
> > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> > 11241 www 1 114 0 61492K 11200K CPU0 0 0:53 78.96% httpd
> > 11237 www 1 110 0 61492K 11276K RUN 1 0:40 66.26% httpd
> > 11238 www 1 108 0 61492K 11276K RUN 0 0:50 56.05% httpd
>
> get syscall trace of high cpu process?
> attach with debugger, find callstacks, let it run a while, get
> callstacks again, see if there are threads that didn't make useful
> progress?
>
I'm not a programmer but I’ll see what i can get. Any information as to
how to do this would be appreciated.
> >
> > Process takes up all resources and is unresponsive, no errors are logged.
>
> happens immediately, gradually, immediately after some unknown event occurs?
>
Pretty much right after i start apache the pids start to consume cpu
time.
Re: apr 1.4.4 breaks mod_jk
Posted by Jeff Trawick <tr...@gmail.com>.
On Tue, May 17, 2011 at 5:33 PM, Mike Jakubik
<mi...@intertainservices.com> wrote:
> Hello,
>
> I have recently discovered while updating my apache22 installation and
> subsequently apr1 from 1.4.2 to 1.4.4 that mod_jk now causes the apache
> server to consume 100% cpu and become unresponsive. I have tried recompiling
> mod_jk with the new apr but that did not help. Below is my environment
> information.
you updated Apache 2.2.x from 2.2.what to 2.2.what?
(you updated apr 1.4.2 to 1.4.4)
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 11241 www 1 114 0 61492K 11200K CPU0 0 0:53 78.96% httpd
> 11237 www 1 110 0 61492K 11276K RUN 1 0:40 66.26% httpd
> 11238 www 1 108 0 61492K 11276K RUN 0 0:50 56.05% httpd
get syscall trace of high cpu process?
attach with debugger, find callstacks, let it run a while, get
callstacks again, see if there are threads that didn't make useful
progress?
>
> Process takes up all resources and is unresponsive, no errors are logged.
happens immediately, gradually, immediately after some unknown event occurs?
> Removing mod_jk from apache solves the problem.
>
> FreeBSD staging.local 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Mar 3 17:32:06
> EST 2011 root@staging.local:/usr/obj/usr/src/sys/STAGING amd64
> Apache/2.2.18 (FreeBSD) mod_jk/1.2.31 PHP/5.3.6 with Suhosin-Patch
> mod_ssl/2.2.18 OpenSSL/0.9.8q
>
> Thanks.
>
>
--
Born in Roswell... married an alien...