You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Wang, Andy" <aw...@ptc.com> on 2014/07/07 17:51:29 UTC
Slow memory leak in mod_jk on Windows
We have a customer that's seeing a very slow memory leak under certain
circumstances that we haven't yet been able to pinpoint. I can
reproduce it, but it requires a very particular method of downloading
files that I don't quite understand yet. (not entirely sure how this
would impact mod_jk either). Our customer is still on an older mod_jk
1.2.37 but I'm able to see the behavior on 1.2.40.
They had Microsoft analyze a memory dump and Microsoft came to the
analysis that mod_jk was leaking 8k at a time and this is the stack of
each allocation:
Function Destination
libapr_1!apr_palloc+212 msvcrt!malloc
libaprutil_1!apr_brigade_create+11 libapr_1!apr_palloc
libhttpd!ap_rflush+19 libaprutil_1!apr_brigade_create
mod_jk+3183a libhttpd!ap_rflush
mod_jk+9729
mod_jk+f3bd
kernel32!TlsSetValueStub
ntdll!_except_handler4
msvcrt!_except_handler4
ntdll!_except_handler4
ntdll!FinalExceptionHandler
msvcrt!_threadstartex
I'm far far from a Windows developer :( so I'm slowly getting a system
configured to do the debugdiag analysis that Microsoft did with the pdb
files to get the actual symbols. This will take me some time so I
thought I'd mail here to see of anyone has seen anything similar or if
there's any thoughts on what would be slowly leaking the 8k through
mod_jk.
If not, I'll try to get the real stack shortly once I get debugdiag
figured out.
Thanks,
Andy
Re: Slow memory leak in mod_jk on Windows
Posted by "Wang, Andy" <aw...@ptc.com>.
I've narrowed this down to JkOptions +FlushPackets
and some combination of workers.properties configurations causing it. I'm trying to pinpoint which combination.
Opened the following bug:
https://issues.apache.org/bugzilla/show_bug.cgi?id=56733
Andy
On Mon, 2014-07-07 at 19:58 +0000, Wang, Andy wrote:
On Mon, 2014-07-07 at 15:51 +0000, Wang, Andy wrote:
> We have a customer that's seeing a very slow memory leak under certain
> circumstances that we haven't yet been able to pinpoint. I can
> reproduce it, but it requires a very particular method of downloading
> files that I don't quite understand yet. (not entirely sure how this
> would impact mod_jk either). Our customer is still on an older mod_jk
> 1.2.37 but I'm able to see the behavior on 1.2.40.
>
> They had Microsoft analyze a memory dump and Microsoft came to the
> analysis that mod_jk was leaking 8k at a time and this is the stack of
> each allocation:
> Function Destination
> libapr_1!apr_palloc+212 msvcrt!malloc
> libaprutil_1!apr_brigade_create+11 libapr_1!apr_palloc
> libhttpd!ap_rflush+19 libaprutil_1!apr_brigade_create
> mod_jk+3183a libhttpd!ap_rflush
> mod_jk+9729
> mod_jk+f3bd
> kernel32!TlsSetValueStub
> ntdll!_except_handler4
> msvcrt!_except_handler4
> ntdll!_except_handler4
> ntdll!FinalExceptionHandler
> msvcrt!_threadstartex
>
> I'm far far from a Windows developer :( so I'm slowly getting a system
> configured to do the debugdiag analysis that Microsoft did with the pdb
> files to get the actual symbols. This will take me some time so I
> thought I'd mail here to see of anyone has seen anything similar or if
> there's any thoughts on what would be slowly leaking the 8k through
> mod_jk.
>
> If not, I'll try to get the real stack shortly once I get debugdiag
> figured out.
>
> Thanks,
> Andy
>
Managed to reproduce this on httpd 2.2.27 and mod_jk 1.2.40
Here's the stack I'm pulling from Microsoft's debugdiag:
Function Destination
libapr_1!allocator_alloc+d6 msvcr100!malloc
libapr_1!apr_palloc+d9 libapr_1!allocator_alloc
libaprutil_1!apr_brigade_create+11 libapr_1!apr_palloc
libhttpd!ap_rflush+19 libaprutil_1!apr_brigade_create
mod_jk!ws_flush+1a libhttpd!ap_rflush
mod_jk!ajp_process_callback+586
mod_jk!ajp_get_reply+c4 mod_jk!ajp_process_callback
mod_jk!ajp_service+60a mod_jk!ajp_get_reply
mod_jk!service+82f
mod_jk!jk_handler+6ea
libhttpd!ap_run_handler+25
libhttpd!ap_invoke_handler+a2 libhttpd!ap_run_handler
libhttpd!ap_process_request+3e libhttpd!ap_invoke_handler
libhttpd!ap_process_http_connection+52 libhttpd!ap_process_request
libhttpd!ap_run_process_connection+25
libhttpd!ap_process_connection+33 libhttpd!ap_run_process_connection
libhttpd!worker_main+a7 libhttpd!ap_process_connection
msvcr100!_callthreadstartex+1b
msvcr100!_threadstartex+64
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+70
ntdll!_RtlUserThreadStart+1b ntdll!__RtlUserThreadStart
msvcr100!_threadstartex
I have a relatively large file I'm downloading through a servlet
(unfortunately I didn't write the servlet and am not completely familiar
with what they're doing) and memory slowly slowly grows and isn't
released. 300mb file grows by a few megs each download.
I can't reproduce with the default tomcat file servlet so not sure what
the deal here is.
Going to poke some more but hoping for some ideas.
Thanks,
Andy
B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB�� [��X��ܚX�KK[XZ[� \�\��][��X��ܚX�P �X�] �\X�K�ܙ�B��܈Y][ۘ[��[X[� �K[XZ[� \�\��Z[ �X�] �\X�K�ܙ�B�
Re: Slow memory leak in mod_jk on Windows
Posted by "Wang, Andy" <aw...@ptc.com>.
On Mon, 2014-07-07 at 15:51 +0000, Wang, Andy wrote:
> We have a customer that's seeing a very slow memory leak under certain
> circumstances that we haven't yet been able to pinpoint. I can
> reproduce it, but it requires a very particular method of downloading
> files that I don't quite understand yet. (not entirely sure how this
> would impact mod_jk either). Our customer is still on an older mod_jk
> 1.2.37 but I'm able to see the behavior on 1.2.40.
>
> They had Microsoft analyze a memory dump and Microsoft came to the
> analysis that mod_jk was leaking 8k at a time and this is the stack of
> each allocation:
> Function Destination
> libapr_1!apr_palloc+212 msvcrt!malloc
> libaprutil_1!apr_brigade_create+11 libapr_1!apr_palloc
> libhttpd!ap_rflush+19 libaprutil_1!apr_brigade_create
> mod_jk+3183a libhttpd!ap_rflush
> mod_jk+9729
> mod_jk+f3bd
> kernel32!TlsSetValueStub
> ntdll!_except_handler4
> msvcrt!_except_handler4
> ntdll!_except_handler4
> ntdll!FinalExceptionHandler
> msvcrt!_threadstartex
>
> I'm far far from a Windows developer :( so I'm slowly getting a system
> configured to do the debugdiag analysis that Microsoft did with the pdb
> files to get the actual symbols. This will take me some time so I
> thought I'd mail here to see of anyone has seen anything similar or if
> there's any thoughts on what would be slowly leaking the 8k through
> mod_jk.
>
> If not, I'll try to get the real stack shortly once I get debugdiag
> figured out.
>
> Thanks,
> Andy
>
Managed to reproduce this on httpd 2.2.27 and mod_jk 1.2.40
Here's the stack I'm pulling from Microsoft's debugdiag:
Function Destination
libapr_1!allocator_alloc+d6 msvcr100!malloc
libapr_1!apr_palloc+d9 libapr_1!allocator_alloc
libaprutil_1!apr_brigade_create+11 libapr_1!apr_palloc
libhttpd!ap_rflush+19 libaprutil_1!apr_brigade_create
mod_jk!ws_flush+1a libhttpd!ap_rflush
mod_jk!ajp_process_callback+586
mod_jk!ajp_get_reply+c4 mod_jk!ajp_process_callback
mod_jk!ajp_service+60a mod_jk!ajp_get_reply
mod_jk!service+82f
mod_jk!jk_handler+6ea
libhttpd!ap_run_handler+25
libhttpd!ap_invoke_handler+a2 libhttpd!ap_run_handler
libhttpd!ap_process_request+3e libhttpd!ap_invoke_handler
libhttpd!ap_process_http_connection+52 libhttpd!ap_process_request
libhttpd!ap_run_process_connection+25
libhttpd!ap_process_connection+33 libhttpd!ap_run_process_connection
libhttpd!worker_main+a7 libhttpd!ap_process_connection
msvcr100!_callthreadstartex+1b
msvcr100!_threadstartex+64
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+70
ntdll!_RtlUserThreadStart+1b ntdll!__RtlUserThreadStart
msvcr100!_threadstartex
I have a relatively large file I'm downloading through a servlet
(unfortunately I didn't write the servlet and am not completely familiar
with what they're doing) and memory slowly slowly grows and isn't
released. 300mb file grows by a few megs each download.
I can't reproduce with the default tomcat file servlet so not sure what
the deal here is.
Going to poke some more but hoping for some ideas.
Thanks,
Andy