You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by JNeuhoff <us...@mhccorp.com> on 2007/01/16 21:33:02 UTC
Apache mod_jk memory leak?
We are running an Apache 2.0.54 , mod_jk 1.2.10 and Tomcat 5.5.17 on a
Windows 2003 server box, average web traffic to Tomcat is about 10 to 20
concurrent HTTP sessions (idle session timeout 15 minutes). After a weekend
of sudden heavy web traffic with up to 150 simultaneous HTTP sessions we
experienced 2 server crashes within 24 hours.
Even though our Tomcat only ever used 20% of its assigned maximum 512MB JRE
Memory and never crashed, mod_jk produced loads of error messages along the
lines:
<<<
[Sun Jan 14 21:25:37 2007] [info] jk_handler::mod_jk.c (1975): Service
error=0 for worker=ajp13
[Sun Jan 14 21:25:40 2007] [error]
ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
returned -54 with errno=54
[Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
(1220): Error sending request on a fresh connection
[Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
Sending request to tomcat failed, recoverable operation attempt=1
[Sun Jan 14 21:25:40 2007] [error]
ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
returned -54 with errno=54
[Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
(1220): Error sending request on a fresh connection
[Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
Sending request to tomcat failed, recoverable operation attempt=2
[Sun Jan 14 21:25:40 2007] [error]
ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
returned -54 with errno=54
[Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
(1220): Error sending request on a fresh connection
[Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
Sending request to tomcat failed, recoverable operation attempt=3
[Sun Jan 14 21:25:40 2007] [error] ajp_service::jk_ajp_common.c (1733):
Error connecting to tomcat. Tomcat is probably not started or is
listening on the wrong port. worker=ajp13 failed
>>>
I also noticed that one of the 2 Apache.exe processes went up in memory
usage from 12MB to a 160MB before Apache failed to serve any requests, even
though both Apche and Tomcat still continued to log incoming requests.
It seems that mod_jk doesn't release the memory for unused TCP-connection
sockets involving port 8009 (the one for the ajp13 protocol). Not even a
restart of the Apache and Tomcat5 NT services cleared up the pipe, they
still refused to serve requests. So I finally ended up rebooting the whole
Windows 2003 server box.
Is there a memory leak problem with Apache 2 / mod_jk ? Any help
appreciated! How can I configure the system to cope with heavier traffic
and, if necessary, tell the user that the system is too busy and to come
back later?
In case it's of any importance, here is the relevant workers.properties for
the ajp13 we are using:
<<<
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
>>>
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8398261
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
BTW: Tomcat (which is still running and which I can access via port 8080)
claims this in its manager status report:
Max threads: 40 Min spare threads: 0 Max spare threads: 0 Current thread
count: 14 Current thread busy: 1 Keeped alive sockets count: 0
Max processing time: 3109 ms Processing time: 141.402 s Request count: 386
Error count: 49 Bytes received: 0.00 MB Bytes sent: 5.14 MB
with 14 corresponding connections in the Ready stage. Whereas the netstat -a
command says there no connections between Tomcat and Apache (see my previous
post).
J.Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8449222
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Apache mod_jk memory leak?
Posted by Per Johnsson <Pe...@omxgroup.com>.
Please do!
It will be much appriciated...
/per jonsson
-----Original Message-----
From: RChartier [mailto:chartier.romain@gmail.com]
Sent: den 14 februari 2007 13:10
To: users@tomcat.apache.org
Subject: RE: Apache mod_jk memory leak?
Hi!
I don't know IIS but maybe the patch could help.
This is not an official patch. WindowsServer2003-KB931311 can be
retrieved by calling microsoft support.
But I can also send you the patch if you can't get it.
Romain
Per Johnsson wrote:
>
> Hi!
> I wonder if this is the same problem we have, but we are using IIS6
> and isapi_redirect.dll and if I'm not totally wrong the same codebase
> is used for apache and iis.
>
> We have come to the conclusion that there is a error in the "free
> memory" causing a crash when the pool workers of the iis is unloaded,
> unfortunaly we have not managed to see what is causing the error.
>
> Am I wrong or is this a Win 2003 Server error which can be helped with
> the suggested patch.
>
> I wrote a longer mail about our problems:
>
> http://marc.theaimsgroup.com/?l=tomcat-user&m=117092232007880&w=2
>
>
> /Per Jonsson
>
> -----Original Message-----
> From: RChartier [mailto:chartier.romain@gmail.com]
> Sent: den 9 februari 2007 16:00
> To: users@tomcat.apache.org
> Subject: Re: [OT] Apache mod_jk memory leak?
>
>
> Hi,
>
> I am using Apache 2.0.59 and Tomcat 5.0.28 with mod_jk 1.2.20 on
> Windows
> 2003 SP1 but the system have some issue on the socket layer. In fact,
> after some time, some users are able to crash our architecture: apache
> answers to request such as *.html but mod_jk says it cannot connect to
> tomcat. Even a restart of services doesn't change anythink. Tomcat is
> running and answer for simple "hello world" jsp pages but cannot
> answer to /manage/html.
>
> I have exacltly the same error logs and thread dumps as you are.
>
> I patch windows with KB 931311 and it seems to work properly.
>
>
> JNeuhoff wrote:
>>
>> I have just managed to repeat the error. 2 of us, from 2 different
>> workstations, hammered our website for a minute, by rapidly clicking
>> on links within the same site before it ended up always responding
>> with a standard Error 503 (Service unavailable) coming from the
>> Apache
> frontend.
>> I checked all the log files, and again the only errors recorded were
>> found in the mod_jk.log:
>>
>> ....
>> [Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142):
>> Service error=0 for worker=ajp13 [Fri Jan 19 11:34:42 2007]
>> [0404:2388] [error] jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 11:34:42 2007] [0404:2388]
>> [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
>> [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c
(1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=2 [Fri Jan 19 11:34:42 2007] [0404:2388] [error]
> jk_ajp_common.c (1928):
>> (ajp13) Connecting to tomcat failed. Tomcat is probably not started
>> or
>
>> is listening on the wrong port ....
>>
>>
>> The 2 connections between Apache and Tomcat according the the netstat
>> -a command just after the web server started the 503 (Service
>> unavilable) responses (hightlighted in bold):
>>
>> Proto Local Address Foreign Address State
>> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
>> TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
>> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306
ESTABLISHED
>> TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap
ESTABLISHED
>> TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
>> TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap
ESTABLISHED
>> TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap
ESTABLISHED
>> TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
>> ESTABLISHED
>> TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds
> TIME_WAIT
>> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
>> TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>>
>>
>> Half another later, long after the connection_pool_timeout,
>> connect_timeout and prepost_timeout, the netstat -a command still
>> shows
>> this:
>>
>> Proto Local Address Foreign Address State
>> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306
ESTABLISHED
>> TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap
ESTABLISHED
>> TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
>> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
>> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>>
>>
>> Yet if I then try again to access the web service running on Tomcat,
>> mod_jk fails to re-establish a connection, according to the
> mod_jk.log:
>>
>> [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 12:50:51 2007] [0404:2404]
>> [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=0
>> [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c
(1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=1 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
> jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 12:50:52 2007] [0404:2404]
>> [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
>> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c
(1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=2 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
> jk_ajp_common.c (1928):
>> (ajp13) Connecting to tomcat failed. Tomcat is probably not started
>> or
>
>> is listening on the wrong port [Fri Jan 19 12:50:52 2007] ajp13
>> daohprw2-a 0.281333 GET /ohpr/servlet/
>> HTTP/1.1 200
>> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142):
>> Service error=0 for worker=ajp13
>>
>> Only Apache continues to respond with a standard 503 error according
>> to its access.log:
>>
>> daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET
>> /ohpr/servlet/ HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U;
>> Windows NT 5.1; en-US;
>> rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
>>
>> In short: There were only 2 established connections, Tomcat was
>> running all the time, and, as expected, only saw 2 active
>> HTTPSessions, yet the rapid hammering on weblinks within the website
>> somehow blocked the connections between Apache and Tomcat. And now
>> Tomcat keeps just listening on port 8009, but mod_jk is unable to
>> establish any TCP connections between Apache and Tomcat. Only a
>> complete server machine re-boot gets things back to normal.
>>
>> Any ideas what might have gone wrong? I am still using this
>> workers.properties:
>>
>> # Define 1 real worker using ajp13
>> worker.list=ajp13
>> # Set properties for worker1 (ajp13)
>> worker.ajp13.type=ajp13
>> worker.ajp13.host=localhost
>> worker.ajp13.port=8009
>> worker.ajp13.connection_pool_timeout=600
>> worker.ajp13.connection_pool_minsize=10
>> worker.ajp13.connect_timeout=15000
>> worker.ajp13.prepost_timeout=10000
>>
>>
>> Regards
>>
>> Juergen Neuhoff
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a88871
> 44 Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe,
> e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> **********************************************************************
> ********** This e-mail and the information it contains may be
> privileged and/or confidential. It is for the intended addressee(s)
> only.
> The unauthorised use, disclosure or copying of this e-mail, or any
> information it contains, is prohibited.
> If you are not an intended recipient, please contact the sender and
> delete the material from your computer.
> **********************************************************************
> **********
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe,
> e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context:
http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8963752
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
********************************************************************************
This e-mail and the information it contains may be privileged and/or
confidential. It is for the intended addressee(s) only.
The unauthorised use, disclosure or copying of this e-mail, or any information it contains, is prohibited.
If you are not an intended recipient, please contact the sender and delete the material from your computer.
********************************************************************************
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Apache mod_jk memory leak?
Posted by RChartier <ch...@gmail.com>.
Hi!
I don't know IIS but maybe the patch could help.
This is not an official patch. WindowsServer2003-KB931311 can be retrieved
by calling microsoft support.
But I can also send you the patch if you can't get it.
Romain
Per Johnsson wrote:
>
> Hi!
> I wonder if this is the same problem we have, but we are using IIS6 and
> isapi_redirect.dll and if I'm not totally wrong the same codebase is
> used for apache and iis.
>
> We have come to the conclusion that there is a error in the "free
> memory" causing a crash when the pool workers of the iis is unloaded,
> unfortunaly we have not managed to see what is causing the error.
>
> Am I wrong or is this a Win 2003 Server error which can be helped with
> the suggested patch.
>
> I wrote a longer mail about our problems:
>
> http://marc.theaimsgroup.com/?l=tomcat-user&m=117092232007880&w=2
>
>
> /Per Jonsson
>
> -----Original Message-----
> From: RChartier [mailto:chartier.romain@gmail.com]
> Sent: den 9 februari 2007 16:00
> To: users@tomcat.apache.org
> Subject: Re: [OT] Apache mod_jk memory leak?
>
>
> Hi,
>
> I am using Apache 2.0.59 and Tomcat 5.0.28 with mod_jk 1.2.20 on Windows
> 2003 SP1 but the system have some issue on the socket layer. In fact,
> after some time, some users are able to crash our architecture: apache
> answers to request such as *.html but mod_jk says it cannot connect to
> tomcat. Even a restart of services doesn't change anythink. Tomcat is
> running and answer for simple "hello world" jsp pages but cannot answer
> to /manage/html.
>
> I have exacltly the same error logs and thread dumps as you are.
>
> I patch windows with KB 931311 and it seems to work properly.
>
>
> JNeuhoff wrote:
>>
>> I have just managed to repeat the error. 2 of us, from 2 different
>> workstations, hammered our website for a minute, by rapidly clicking
>> on links within the same site before it ended up always responding
>> with a standard Error 503 (Service unavailable) coming from the Apache
> frontend.
>> I checked all the log files, and again the only errors recorded were
>> found in the mod_jk.log:
>>
>> ....
>> [Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142):
>> Service error=0 for worker=ajp13 [Fri Jan 19 11:34:42 2007]
>> [0404:2388] [error] jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 11:34:42 2007] [0404:2388] [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
>> [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=2 [Fri Jan 19 11:34:42 2007] [0404:2388] [error]
> jk_ajp_common.c (1928):
>> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or
>
>> is listening on the wrong port ....
>>
>>
>> The 2 connections between Apache and Tomcat according the the netstat
>> -a command just after the web server started the 503 (Service
>> unavilable) responses (hightlighted in bold):
>>
>> Proto Local Address Foreign Address State
>> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
>> TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
>> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
>> TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
>> TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
>> ESTABLISHED
>> TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds
> TIME_WAIT
>> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
>> TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn
> TIME_WAIT
>> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>>
>>
>> Half another later, long after the connection_pool_timeout,
>> connect_timeout and prepost_timeout, the netstat -a command still
>> shows
>> this:
>>
>> Proto Local Address Foreign Address State
>> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
>> TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds
>> TIME_WAIT
>> TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED
>> TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
>> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
>> TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
>> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>>
>>
>> Yet if I then try again to access the web service running on Tomcat,
>> mod_jk fails to re-establish a connection, according to the
> mod_jk.log:
>>
>> [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 12:50:51 2007] [0404:2404] [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=0
>> [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=1 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
> jk_ajp_common.c (948):
>> (ajp13) can't receive the response message from tomcat, network
>> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
>> 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
>> (ajp13) Tomcat is down or refused connection. No response has been
>> sent to the client (yet) [Fri Jan 19 12:50:52 2007] [0404:2404] [info]
>
>> jk_ajp_common.c (1877):
>> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
>> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916):
>> (ajp13) sending request to tomcat failed, recoverable operation
>> attempt=2 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
> jk_ajp_common.c (1928):
>> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or
>
>> is listening on the wrong port [Fri Jan 19 12:50:52 2007] ajp13
>> daohprw2-a 0.281333 GET /ohpr/servlet/
>> HTTP/1.1 200
>> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142):
>> Service error=0 for worker=ajp13
>>
>> Only Apache continues to respond with a standard 503 error according
>> to its access.log:
>>
>> daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET
>> /ohpr/servlet/ HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U;
>> Windows NT 5.1; en-US;
>> rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
>>
>> In short: There were only 2 established connections, Tomcat was
>> running all the time, and, as expected, only saw 2 active
>> HTTPSessions, yet the rapid hammering on weblinks within the website
>> somehow blocked the connections between Apache and Tomcat. And now
>> Tomcat keeps just listening on port 8009, but mod_jk is unable to
>> establish any TCP connections between Apache and Tomcat. Only a
>> complete server machine re-boot gets things back to normal.
>>
>> Any ideas what might have gone wrong? I am still using this
>> workers.properties:
>>
>> # Define 1 real worker using ajp13
>> worker.list=ajp13
>> # Set properties for worker1 (ajp13)
>> worker.ajp13.type=ajp13
>> worker.ajp13.host=localhost
>> worker.ajp13.port=8009
>> worker.ajp13.connection_pool_timeout=600
>> worker.ajp13.connection_pool_minsize=10
>> worker.ajp13.connect_timeout=15000
>> worker.ajp13.prepost_timeout=10000
>>
>>
>> Regards
>>
>> Juergen Neuhoff
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8887144
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ********************************************************************************
> This e-mail and the information it contains may be privileged and/or
> confidential. It is for the intended addressee(s) only.
> The unauthorised use, disclosure or copying of this e-mail, or any
> information it contains, is prohibited.
> If you are not an intended recipient, please contact the sender and delete
> the material from your computer.
> ********************************************************************************
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8963752
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [OT] Apache mod_jk memory leak?
Posted by Per Johnsson <Pe...@omxgroup.com>.
Hi!
I wonder if this is the same problem we have, but we are using IIS6 and
isapi_redirect.dll and if I'm not totally wrong the same codebase is
used for apache and iis.
We have come to the conclusion that there is a error in the "free
memory" causing a crash when the pool workers of the iis is unloaded,
unfortunaly we have not managed to see what is causing the error.
Am I wrong or is this a Win 2003 Server error which can be helped with
the suggested patch.
I wrote a longer mail about our problems:
http://marc.theaimsgroup.com/?l=tomcat-user&m=117092232007880&w=2
/Per Jonsson
-----Original Message-----
From: RChartier [mailto:chartier.romain@gmail.com]
Sent: den 9 februari 2007 16:00
To: users@tomcat.apache.org
Subject: Re: [OT] Apache mod_jk memory leak?
Hi,
I am using Apache 2.0.59 and Tomcat 5.0.28 with mod_jk 1.2.20 on Windows
2003 SP1 but the system have some issue on the socket layer. In fact,
after some time, some users are able to crash our architecture: apache
answers to request such as *.html but mod_jk says it cannot connect to
tomcat. Even a restart of services doesn't change anythink. Tomcat is
running and answer for simple "hello world" jsp pages but cannot answer
to /manage/html.
I have exacltly the same error logs and thread dumps as you are.
I patch windows with KB 931311 and it seems to work properly.
JNeuhoff wrote:
>
> I have just managed to repeat the error. 2 of us, from 2 different
> workstations, hammered our website for a minute, by rapidly clicking
> on links within the same site before it ended up always responding
> with a standard Error 503 (Service unavailable) coming from the Apache
frontend.
> I checked all the log files, and again the only errors recorded were
> found in the mod_jk.log:
>
> ....
> [Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142):
> Service error=0 for worker=ajp13 [Fri Jan 19 11:34:42 2007]
> [0404:2388] [error] jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network
> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
> 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been
> sent to the client (yet) [Fri Jan 19 11:34:42 2007] [0404:2388] [info]
> jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
> [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation
> attempt=2 [Fri Jan 19 11:34:42 2007] [0404:2388] [error]
jk_ajp_common.c (1928):
> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or
> is listening on the wrong port ....
>
>
> The 2 connections between Apache and Tomcat according the the netstat
> -a command just after the web server started the 503 (Service
> unavilable) responses (hightlighted in bold):
>
> Proto Local Address Foreign Address State
> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
> TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
> TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
> TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
> ESTABLISHED
> TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds
TIME_WAIT
> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
> TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn
TIME_WAIT
> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>
>
> Half another later, long after the connection_pool_timeout,
> connect_timeout and prepost_timeout, the netstat -a command still
> shows
> this:
>
> Proto Local Address Foreign Address State
> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
> TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>
>
> Yet if I then try again to access the web service running on Tomcat,
> mod_jk fails to re-establish a connection, according to the
mod_jk.log:
>
> [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network
> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
> 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been
> sent to the client (yet) [Fri Jan 19 12:50:51 2007] [0404:2404] [info]
> jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=0
> [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation
> attempt=1 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network
> problems or tomcat is down (127.0.0.1:8009), err=-53 [Fri Jan 19
> 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been
> sent to the client (yet) [Fri Jan 19 12:50:52 2007] [0404:2404] [info]
> jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation
> attempt=2 [Fri Jan 19 12:50:52 2007] [0404:2404] [error]
jk_ajp_common.c (1928):
> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or
> is listening on the wrong port [Fri Jan 19 12:50:52 2007] ajp13
> daohprw2-a 0.281333 GET /ohpr/servlet/
> HTTP/1.1 200
> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142):
> Service error=0 for worker=ajp13
>
> Only Apache continues to respond with a standard 503 error according
> to its access.log:
>
> daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET
> /ohpr/servlet/ HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U;
> Windows NT 5.1; en-US;
> rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
>
> In short: There were only 2 established connections, Tomcat was
> running all the time, and, as expected, only saw 2 active
> HTTPSessions, yet the rapid hammering on weblinks within the website
> somehow blocked the connections between Apache and Tomcat. And now
> Tomcat keeps just listening on port 8009, but mod_jk is unable to
> establish any TCP connections between Apache and Tomcat. Only a
> complete server machine re-boot gets things back to normal.
>
> Any ideas what might have gone wrong? I am still using this
> workers.properties:
>
> # Define 1 real worker using ajp13
> worker.list=ajp13
> # Set properties for worker1 (ajp13)
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
> worker.ajp13.connection_pool_timeout=600
> worker.ajp13.connection_pool_minsize=10
> worker.ajp13.connect_timeout=15000
> worker.ajp13.prepost_timeout=10000
>
>
> Regards
>
> Juergen Neuhoff
>
>
--
View this message in context:
http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8887144
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
********************************************************************************
This e-mail and the information it contains may be privileged and/or
confidential. It is for the intended addressee(s) only.
The unauthorised use, disclosure or copying of this e-mail, or any information it contains, is prohibited.
If you are not an intended recipient, please contact the sender and delete the material from your computer.
********************************************************************************
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Apache mod_jk memory leak?
Posted by RChartier <ch...@gmail.com>.
Hi,
I am using Apache 2.0.59 and Tomcat 5.0.28 with mod_jk 1.2.20 on Windows
2003 SP1 but the system have some issue on the socket layer. In fact, after
some time, some users are able to crash our architecture: apache answers to
request such as *.html but mod_jk says it cannot connect to tomcat. Even a
restart of services doesn't change anythink. Tomcat is running and answer
for simple "hello world" jsp pages but cannot answer to /manage/html.
I have exacltly the same error logs and thread dumps as you are.
I patch windows with KB 931311 and it seems to work properly.
JNeuhoff wrote:
>
> I have just managed to repeat the error. 2 of us, from 2 different
> workstations, hammered our website for a minute, by rapidly clicking on
> links within the same site before it ended up always responding with a
> standard Error 503 (Service unavailable) coming from the Apache frontend.
> I checked all the log files, and again the only errors recorded were found
> in the mod_jk.log:
>
> ....
> [Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142): Service
> error=0 for worker=ajp13
> [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network problems
> or tomcat is down (127.0.0.1:8009), err=-53
> [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been sent to
> the client (yet)
> [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
> [Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation attempt=2
> [Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1928):
> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
> listening on the wrong port
> ....
>
>
> The 2 connections between Apache and Tomcat according the the netstat -a
> command just after the web server started the 503 (Service unavilable)
> responses (hightlighted in bold):
>
> Proto Local Address Foreign Address State
> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
> TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
> TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
> TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
> ESTABLISHED
> TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds TIME_WAIT
> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
> TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>
>
> Half another later, long after the connection_pool_timeout,
> connect_timeout and prepost_timeout, the netstat -a command still shows
> this:
>
> Proto Local Address Foreign Address State
> TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
> TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds
> TIME_WAIT
> TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED
> TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
> TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
> TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
> TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
>
>
> Yet if I then try again to access the web service running on Tomcat,
> mod_jk fails to re-establish a connection, according to the mod_jk.log:
>
> [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network problems
> or tomcat is down (127.0.0.1:8009), err=-53
> [Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been sent to
> the client (yet)
> [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=0
> [Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation attempt=1
> [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (948):
> (ajp13) can't receive the response message from tomcat, network problems
> or tomcat is down (127.0.0.1:8009), err=-53
> [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
> (ajp13) Tomcat is down or refused connection. No response has been sent to
> the client (yet)
> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1877):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916):
> (ajp13) sending request to tomcat failed, recoverable operation attempt=2
> [Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1928):
> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
> listening on the wrong port
> [Fri Jan 19 12:50:52 2007] ajp13 daohprw2-a 0.281333 GET /ohpr/servlet/
> HTTP/1.1 200
> [Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142): Service
> error=0 for worker=ajp13
>
> Only Apache continues to respond with a standard 503 error according to
> its access.log:
>
> daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET /ohpr/servlet/
> HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
> rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
>
> In short: There were only 2 established connections, Tomcat was running
> all the time, and, as expected, only saw 2 active HTTPSessions, yet the
> rapid hammering on weblinks within the website somehow blocked the
> connections between Apache and Tomcat. And now Tomcat keeps just listening
> on port 8009, but mod_jk is unable to establish any TCP connections
> between Apache and Tomcat. Only a complete server machine re-boot gets
> things back to normal.
>
> Any ideas what might have gone wrong? I am still using this
> workers.properties:
>
> # Define 1 real worker using ajp13
> worker.list=ajp13
> # Set properties for worker1 (ajp13)
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
> worker.ajp13.connection_pool_timeout=600
> worker.ajp13.connection_pool_minsize=10
> worker.ajp13.connect_timeout=15000
> worker.ajp13.prepost_timeout=10000
>
>
> Regards
>
> Juergen Neuhoff
>
>
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8887144
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
I have just managed to repeat the error. 2 of us, from 2 different
workstations, hammered our website for a minute, by rapidly clicking on
links within the same site before it ended up always responding with a
standard Error 503 (Service unavailable) coming from the Apache frontend. I
checked all the log files, and again the only errors recorded were found in
the mod_jk.log:
....
[Fri Jan 19 11:34:42 2007] [0404:2392] [info] mod_jk.c (2142): Service
error=0 for worker=ajp13
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 11:34:42 2007] [0404:2388] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=2
[Fri Jan 19 11:34:42 2007] [0404:2388] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
....
The 2 connections between Apache and Tomcat according the the netstat -a
command just after the web server started the 503 (Service unavilable)
responses (hightlighted in bold):
Proto Local Address Foreign Address State
TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2093 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
TCP DAOHPRW2:2110 DAOHPRW2.essex.ac.uk:8009 TIME_WAIT
TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
TCP DAOHPRW2:2077 daohprp2.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2176 sernt29.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2177 sernt29.essex.ac.uk:1025 ESTABLISHED
TCP DAOHPRW2:2181 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2194 sernt50.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2204 sernt5.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2215 sernt25.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2230 dalinux6a.essex.ac.uk:netbios-ssn
ESTABLISHED
TCP DAOHPRW2:2250 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2253 bodger.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:http daohpr1.essex.ac.uk:1528 TIME_WAIT
TCP DAOHPRW2:http daohpr2.essex.ac.uk:1936 TIME_WAIT
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2232 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2237 sernt9.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2245 serlx08.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2249 sernt2.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:2261 sernt23.essex.ac.uk:netbios-ssn TIME_WAIT
TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
Half another later, long after the connection_pool_timeout, connect_timeout
and prepost_timeout, the netstat -a command still shows this:
Proto Local Address Foreign Address State
TCP DAOHPRW2:epmap DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:1078 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:3389 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8080 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:10000 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:8005 DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:2076 daohprp2.essex.ac.uk:3306 ESTABLISHED
TCP DAOHPRW2:2754 sernt29.essex.ac.uk:microsoft-ds TIME_WAIT
TCP DAOHPRW2:2782 sernt29.essex.ac.uk:epmap ESTABLISHED
TCP DAOHPRW2:2783 sernt29.essex.ac.uk:1025 ESTABLISHED
TCP DAOHPRW2:http DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:netbios-ssn DAOHPRW2.essex.ac.uk:0 LISTENING
TCP DAOHPRW2:microsoft-ds daohpr1.essex.ac.uk:1541 ESTABLISHED
TCP DAOHPRW2:3389 daohpr1.essex.ac.uk:1533 ESTABLISHED
Yet if I then try again to access the web service running on Tomcat, mod_jk
fails to re-establish a connection, according to the mod_jk.log:
[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:51 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=0
[Fri Jan 19 12:50:51 2007] [0404:2404] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (948):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1566):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1877):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] jk_ajp_common.c (1916):
(ajp13) sending request to tomcat failed, recoverable operation attempt=2
[Fri Jan 19 12:50:52 2007] [0404:2404] [error] jk_ajp_common.c (1928):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
[Fri Jan 19 12:50:52 2007] ajp13 daohprw2-a 0.281333 GET /ohpr/servlet/
HTTP/1.1 200
[Fri Jan 19 12:50:52 2007] [0404:2404] [info] mod_jk.c (2142): Service
error=0 for worker=ajp13
Only Apache continues to respond with a standard 503 error according to its
access.log:
daohpr1.essex.ac.uk - - [19/Jan/2007:12:50:51 +0000] "GET /ohpr/servlet/
HTTP/1.1" 503 1101 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9"
In short: There were only 2 established connections, Tomcat was running all
the time, and, as expected, only saw 2 active HTTPSessions, yet the rapid
hammering on weblinks within the website somehow blocked the connections
between Apache and Tomcat. And now Tomcat keeps just listening on port 8009,
but mod_jk is unable to establish any TCP connections between Apache and
Tomcat. Only a complete server machine re-boot gets things back to normal.
Any ideas what might have gone wrong? I am still using this
workers.properties:
# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connection_pool_minsize=10
worker.ajp13.connect_timeout=15000
worker.ajp13.prepost_timeout=10000
Regards
Juergen Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8448796
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
Hello Chris,
> Check to see if these sessions ever go away. There is a bug in TC 5.5 up
> to and including 5.5.20 (could be 5.0 as well, I'm not sure) where
> sessions can live forever under load. If you are load testing to check
> out the memory usage, you might be triggering this bug where sessions
> are not expired.
>
> (http://issues.apache.org/bugzilla/show_bug.cgi?id=37356)
Our http sessions work fine with Tomcat 5.5.17. Both the Tomcat manager and
my own session listener which I wrote to keep track of the sessions confirm
that all sessions time out after 15 minutes which is the value I set them to
timeout in the web.xml of our servlet. So no, this particular bug seems to
be fixed.
Juergen Neuhoff
PS.: To Rainer Jung:
Shall I put the problem with the mod_jk connection problems as a bug entry
into bugzilla? Our web service clearly has gone into an unrecoverable state
after just 2 users hit our webservice for just one minute with a rapid
successions of keyborad hits in their respective web browsers! I still
haven't re-booted it in case more information is needed.
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8452758
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Juergen,
JNeuhoff wrote:
> Is this also the reason for an occasional rapid increase in the number of
> HTTPSessions? I noticed that at times Tomcat reports 3 to 4 times more
> active sessions than there were users (as defined by their IP-address and
> web browser's user agent string).
Check to see if these sessions ever go away. There is a bug in TC 5.5 up
to and including 5.5.20 (could be 5.0 as well, I'm not sure) where
sessions can live forever under load. If you are load testing to check
out the memory usage, you might be triggering this bug where sessions
are not expired.
(http://issues.apache.org/bugzilla/show_bug.cgi?id=37356)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFsPV89CaO5/Lv0PARAoEdAKCeonCKv/NmKSg/JP7rYZJudQHMBgCdG2T1
VH0pF2BkJZm9O4f87BrooXo=
=SCxx
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
> > Yes, it starts out with a much smaller memory, around 10 to 15 MB or so,
> > even after a few initial connections to Tomcat from one user session.
> >
> I see, any idea, how the delta 70-15=55MB relates to connections (if you
> do stress tests with real parallelity e.g. 20, 50, 100, 200, how does
> memory behave after stopping the load)?
It seems to grow linearly with the increasing number of connections.
I'd have to run different setups in the stresstest to see whether the amount
of data traffic over a single connection has a similar effect.
> These ones are OK, they are info level and they only tell you, that when
> mod_jk tried to send back answers, it found out, that the browser
> already closed the connection. That's true, because every reload during
> waiting for response in the browser will reset the browser connection to
> the web server. This will only be detected, when mod_jk tries to use the
> connection to either read parts of the request (this normally already
> succeeded when you did the reload) or it tries to write back parts of
> the result.
Is this also the reason for an occasional rapid increase in the number of
HTTPSessions? I noticed that at times Tomcat reports 3 to 4 times more
active sessions than there were users (as defined by their IP-address and
web browser's user agent string).
> Those err=-53 are bad, 53 on Windows is connection aborted. You should
> find something about them in the tomcat logs.
I checked all the usual log files (stdout*.log, stderr*.log,
localhost*.log,mod_jk.log) from Tomcat as well as the access.log from
Apache. The only one with error messages was in mod_jk.log. There were no
excessive numbers of connections, in fact just 2 of them! I will run some
more stresstests to find out more. Would a socket_timeout help flush and/or
release a blocked connection? I am also thinking about setting the
connection_pool_minsize to zero.
> Your statement about 2 connections left but nevertheless a server reboot
> was needed is unclear to me. I would expect, that at most a tomcat
> restart would be needed. Could be, that you'll find an OutOfMemoryError
> in your tomcat logs (because of the huge amount of parallel unfinished
> requests), which might have sent tomcat to an undefined state.
There were no OutOfMemoryError in Tomcat, in fact its memory usage was never
more than 150MB, its JRE is configured to a maximum 512MB. After the traffic
surge was over, the Apache/mod_jk/Tomcat never recovered, not even after the
HTTPSesison idle timeout (set to 15 minutes). I'll try and reproduce the
same error condition later on today.
Regards
Juergen Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8446606
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Rainer Jung <ra...@kippdata.de>.
JNeuhoff wrote:
> Yes, it starts out with a much smaller memory, around 10 to 15 MB or so,
> even after a few initial connections to Tomcat from one user session.
>
I see, any idea, how the delta 70-15=55MB relates to connections (if you
do stress tests with real parallelity e.g. 20, 50, 100, 200, how does
memory behave after stopping the load)?
> I just downloaded the file (of the same name) from another mirror, and this
> time it is really version 1.2.20, looks like the other mirror was corrupted
> (same file name, old version).
No idea why, it's good you observed it.
> Anyway, I made another interesting observation for both mod_jk 1.2.10 and
> 1.2.20: When I hammer my website rapidly with requests, by clicking too
> quickly in my web browser on links within the same site, mod_jk eventually
> becomes unable to receive responses from Tomcat, causing Apache2 to come
> with standard Error 503 replies. I was using the workers.properties with
> this:
>
> # Define 1 real worker using ajp13
> worker.list=ajp13
> # Set properties for worker1 (ajp13)
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
> worker.ajp13.connection_pool_timeout=600
> worker.ajp13.connection_pool_minsize=10
> worker.ajp13.connect_timeout=15000
> worker.ajp13.prepost_timeout=10000
>
> And only mod_jk.log had some error messages, first in increasing numbers
> something like this:
>
> ....
> [Thu Jan 18 17:10:20 2007] [3336:0792] [info] jk_ajp_common.c (1410):
> Writing to client aborted or client network problems
> [Thu Jan 18 17:10:20 2007] [3336:0792] [info] jk_ajp_common.c (1795):
> (ajp13) request failed, because of client write error without recovery in
> send loop attempt=0
> ....
These ones are OK, they are info level and they only tell you, that when
mod_jk tried to send back answers, it found out, that the browser
already closed the connection. That's true, because every reload during
waiting for response in the browser will reset the browser connection to
the web server. This will only be detected, when mod_jk tries to use the
connection to either read parts of the request (this normally already
succeeded when you did the reload) or it tries to write back parts of
the result.
> and finally more and more of the
>
> ....
> [Thu Jan 18 17:10:59 2007] [3336:0188] [info] mod_jk.c (2063): Service
> error=0 for worker=ajp13
> [Thu Jan 18 17:11:00 2007] [3336:1368] [error] jk_ajp_common.c (1015):
> (ajp13) can't receive the response message from tomcat, network problems or
> tomcat (127.0.0.1:8009) is down -53
> [Thu Jan 18 17:11:00 2007] [3336:1368] [error] jk_ajp_common.c (1536):
> (ajp13) Tomcat is down or refused connection. No response has been sent to
> the client (yet)
> [Thu Jan 18 17:11:00 2007] [3336:1368] [info] jk_ajp_common.c (1828):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=0
> [Thu Jan 18 17:11:00 2007] [3336:1368] [info] jk_ajp_common.c (1867):
> (ajp13) sending request to tomcat failed, recoverable operation attempt=1
> [Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (947):
> (ajp13) can't receive the response message from tomcat, network problems or
> tomcat is down (127.0.0.1:8009), err=-53
> [Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (1562):
> (ajp13) Tomcat is down or network problems. Part of the response has already
> been sent to the client
> [Thu Jan 18 17:11:00 2007] [3336:0216] [info] jk_ajp_common.c (1828):
> (ajp13) receiving from tomcat failed, recoverable operation attempt=1
> [Thu Jan 18 17:11:00 2007] [3336:0216] [info] jk_ajp_common.c (1867):
> (ajp13) sending request to tomcat failed, recoverable operation attempt=2
> [Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (1879):
> (ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
> listening on the wrong port
> ....
>
Those err=-53 are bad, 53 on Windows is connection aborted. You should
find something about them in the tomcat logs.
> Only a server re-boot cleared up the connections (there were only 2 of them
> established).
>
> Is it possible that mod_jk can't cope with queueing too many rapidly
> incoming requests? Admittedly, this doesn't happen from normal users, but a
> malicious person would be able to bring down the web service this way. Are
> there preventitive settings for this scenario?
It's not really a problem of rapidly incoming, but you demand more
throughput, than the system can provide. If you click 100 times a second
(not waiting for replies) and your system can only deliver 50 responses
per second, then you will queue up unfinished requests (50 after 1
second, 100 after 2, 150 after 3 etc.) Each unfinished request needs a
separate Thread in Apache and in Tomcat, o since you only provide 250 of
them, you can saturate the system quite fast (again assuming your
throughput is lower than the request injection rate).
If you want to prevent denial of service, you need to do that on a low
protocol level. There are a couple of Apache modules, but more likely
you'll use some sort of security appliance. But that would be better
discussed in a separate mail thread.
Your statement about 2 connections left but nevertheless a server reboot
was needed is unclear to me. I would expect, that at most a tomcat
restart would be needed. Could be, that you'll find an OutOfMemoryError
in your tomcat logs (because of the huge amount of parallel unfinished
requests), which might have sent tomcat to an undefined state.
>
> Regards
>
> J.Neuhoff
>
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
> It closes the connections, but it doesn't release all objects related to
> the corresponding cache slot. Somehow I have the feeling, that it's not
> really worth optimizing this, because 70MB for a web server doesn't
> sound that much relative to hardware sizes of the last years. I assume
> you are not moving into the embedded world :)
> I've got no idea, how biig the process would be, if you would send your
> requests just once to make the basic initializations happen. Will it be
> much smaller than 70MB?
Yes, it starts out with a much smaller memory, around 10 to 15 MB or so,
even after a few initial connections to Tomcat from one user session.
> > BTW.: The mod_jk 1.1.20, downloaded from
> >
> http://mirrors.dedipower.com/ftp.apache.org/tomcat/tomcat-connectors/jk/binaries/win32/jk-1.2.20/mod_jk->
> apache-2.0.58.so
> > identifies itself as mod_jk/1.2.19, not as mod_jk/1.2.20 !
>
> I didn't test (I'm rarely on Win), but I had a look at the binary:
>
> - it's the same as on the origin server (downloaded a minute ago)
> - it has 1.2.20 as a version string inside and no 1.2.19
> - it's md5 checksum is:
>
I just downloaded the file (of the same name) from another mirror, and this
time it is really version 1.2.20, looks like the other mirror was corrupted
(same file name, old version).
Anyway, I made another interesting observation for both mod_jk 1.2.10 and
1.2.20: When I hammer my website rapidly with requests, by clicking too
quickly in my web browser on links within the same site, mod_jk eventually
becomes unable to receive responses from Tomcat, causing Apache2 to come
with standard Error 503 replies. I was using the workers.properties with
this:
# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connection_pool_minsize=10
worker.ajp13.connect_timeout=15000
worker.ajp13.prepost_timeout=10000
And only mod_jk.log had some error messages, first in increasing numbers
something like this:
....
[Thu Jan 18 17:10:20 2007] [3336:0792] [info] jk_ajp_common.c (1410):
Writing to client aborted or client network problems
[Thu Jan 18 17:10:20 2007] [3336:0792] [info] jk_ajp_common.c (1795):
(ajp13) request failed, because of client write error without recovery in
send loop attempt=0
....
and finally more and more of the
....
[Thu Jan 18 17:10:59 2007] [3336:0188] [info] mod_jk.c (2063): Service
error=0 for worker=ajp13
[Thu Jan 18 17:11:00 2007] [3336:1368] [error] jk_ajp_common.c (1015):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat (127.0.0.1:8009) is down -53
[Thu Jan 18 17:11:00 2007] [3336:1368] [error] jk_ajp_common.c (1536):
(ajp13) Tomcat is down or refused connection. No response has been sent to
the client (yet)
[Thu Jan 18 17:11:00 2007] [3336:1368] [info] jk_ajp_common.c (1828):
(ajp13) receiving from tomcat failed, recoverable operation attempt=0
[Thu Jan 18 17:11:00 2007] [3336:1368] [info] jk_ajp_common.c (1867):
(ajp13) sending request to tomcat failed, recoverable operation attempt=1
[Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (947):
(ajp13) can't receive the response message from tomcat, network problems or
tomcat is down (127.0.0.1:8009), err=-53
[Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (1562):
(ajp13) Tomcat is down or network problems. Part of the response has already
been sent to the client
[Thu Jan 18 17:11:00 2007] [3336:0216] [info] jk_ajp_common.c (1828):
(ajp13) receiving from tomcat failed, recoverable operation attempt=1
[Thu Jan 18 17:11:00 2007] [3336:0216] [info] jk_ajp_common.c (1867):
(ajp13) sending request to tomcat failed, recoverable operation attempt=2
[Thu Jan 18 17:11:00 2007] [3336:0216] [error] jk_ajp_common.c (1879):
(ajp13) Connecting to tomcat failed. Tomcat is probably not started or is
listening on the wrong port
....
Only a server re-boot cleared up the connections (there were only 2 of them
established).
Is it possible that mod_jk can't cope with queueing too many rapidly
incoming requests? Admittedly, this doesn't happen from normal users, but a
malicious person would be able to bring down the web service this way. Are
there preventitive settings for this scenario?
Regards
J.Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8436127
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Rainer Jung <ra...@kippdata.de>.
JNeuhoff wrote:
> I have completed some stresstests (with up to 500 concurrent users) on Apache
> 2.0.59, mod_jk 2.1.20, Tomcat 5.5.17, using the following
> workers.properties:
>
...
>
> and it seems to be coping just fine now under the stressload. I think I'll
> use this setup on our live server now. Thanks a lot for your help.
Good to know.
>
> Just one final question:
>
> Despite my connection_pool_timeout=600 and connection_pool_minsize=10, one
> of the Apache2.exe processes remains at 70MB memory usage, even after the
> connection pool has gone down to 10 connections (verified by a netstat
> call). The good news is, the memory doesn't seem grow any further though.
> Still, keeping the 70MB for just the minimum 10 connections is a lot of
> cache memory. Maybe the connection pool releases timed out connections as
> expected, but doesn't flush them?
It closes the connections, but it doesn't release all objects related to
the corresponding cache slot. Somehow I have the feeling, that it's not
really worth optimizing this, because 70MB for a web server doesn't
sound that much relative to hardware sizes of the last years. I assume
you are not moving into the embedded world :)
I've got no idea, how biig the process would be, if you would send your
requests just once to make the basic initializations happen. Will it be
much smaller than 70MB?
> BTW.: The mod_jk 1.1.20, downloaded from
> http://mirrors.dedipower.com/ftp.apache.org/tomcat/tomcat-connectors/jk/binaries/win32/jk-1.2.20/mod_jk-apache-2.0.58.so
> identifies itself as mod_jk/1.2.19, not as mod_jk/1.2.20 !
I didn't test (I'm rarely on Win), but I had a look at the binary:
- it's the same as on the origin server (downloaded a minute ago)
- it has 1.2.20 as a version string inside and no 1.2.19
- it's md5 checksum is:
MD5 (mod_jk-apache-2.0.58.so) = e0bbc07e656c17fa3bd4fd4ea05f457a
You should check, if this really is the file you are actually using.
Maybe I didn't understand correctly, what your idea of "identifiying" is.
> Regards
>
> J.Neuhoff
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
I have completed some stresstests (with up to 500 concurrent users) on Apache
2.0.59, mod_jk 2.1.20, Tomcat 5.5.17, using the following
workers.properties:
<<<
# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connection_pool_minsize=10
worker.ajp13.connect_timeout=15000
worker.ajp13.prepost_timeout=10000
>>>
and this in the httpd.conf:
<<<
....
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_winnt.c>
ThreadsPerChild 250
MaxRequestsPerChild 0
</IfModule>
....
>>>
and it seems to be coping just fine now under the stressload. I think I'll
use this setup on our live server now. Thanks a lot for your help.
Just one final question:
Despite my connection_pool_timeout=600 and connection_pool_minsize=10, one
of the Apache2.exe processes remains at 70MB memory usage, even after the
connection pool has gone down to 10 connections (verified by a netstat
call). The good news is, the memory doesn't seem grow any further though.
Still, keeping the 70MB for just the minimum 10 connections is a lot of
cache memory. Maybe the connection pool releases timed out connections as
expected, but doesn't flush them?
BTW.: The mod_jk 1.1.20, downloaded from
http://mirrors.dedipower.com/ftp.apache.org/tomcat/tomcat-connectors/jk/binaries/win32/jk-1.2.20/mod_jk-apache-2.0.58.so
identifies itself as mod_jk/1.2.19, not as mod_jk/1.2.20 !
Regards
J.Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8428869
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Rainer Jung <ra...@kippdata.de>.
JNeuhoff wrote:
>> By default apache will use as max number of connections the same value,
>> as the max number of threads, so that each thread can get it's own
>> connection. By default it will shrink the connection pool down to half
>> of the max size. There is a min value you can configure if this doesn't
>> fit your needs.
>
> How do I set this min value? Is it the MinSpareThreads directive?
>
Please study the docs. In the workers.properties description of the
reference guide, you will find the params to configure the connection cache.
>> Yes, the Windows MPM has one observer process and a second one, that
>> handles all the requests. If you are really low on server ressources,
>> you can shrink you connection pool from 250 (equals number of threads
>> you configured) to 200 and set the minimum to something like 10. You
>> should also consider a connect_timeout (e.g. 15000) and a
>> prepost_timeout (e.g. 10000).
>
> Does mod_jk use 2 threads (2 socket connections) for each HTTPSession in
> Tomcat, as seemed to be indicated by the 'netstat -a' listing?
No, it's normal that you find a local connection twice in netstat, once
from the point of view of each end of the connection.
>
> Regards
>
> J.Neuhoff
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
> By default apache will use as max number of connections the same value,
> as the max number of threads, so that each thread can get it's own
> connection. By default it will shrink the connection pool down to half
> of the max size. There is a min value you can configure if this doesn't
> fit your needs.
How do I set this min value? Is it the MinSpareThreads directive?
> Yes, the Windows MPM has one observer process and a second one, that
> handles all the requests. If you are really low on server ressources,
> you can shrink you connection pool from 250 (equals number of threads
> you configured) to 200 and set the minimum to something like 10. You
> should also consider a connect_timeout (e.g. 15000) and a
> prepost_timeout (e.g. 10000).
Does mod_jk use 2 threads (2 socket connections) for each HTTPSession in
Tomcat, as seemed to be indicated by the 'netstat -a' listing?
Regards
J.Neuhoff
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8410578
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Rainer Jung <ra...@kippdata.de>.
JNeuhoff wrote:
> Yesterday, I indeed upgraded another server box to Apache 2.0.59, mod_jk
> 1.2.20, Tomcat 5.5.17, and also set the connection_pool_timeout=600 (is that
> value 600 seconds, or milliseconds?). I then subjected this server to a
> brief stresstest (roughly 50 simultanious HTTP sessions on Tomcat's end) and
> noticed the following:
Good, the value is in seconds, and the info has been added to the docs
for 1.2.21 a few days ago and to the online docs also :) Most other
timeouts are milliseconds though, but for the other ones the correct
info is already in the docs.
> Of the two 'Apache.exe' processes running under the Apache2 NT service, one
> grew in memory usage quickly from an initial 12MB to nearly 65MB, and stayed
> at that level overnight, even though there were no further requests made to
> that server since yesterday. I also noticed that a simple 'netstat -a'
> reported 35 established connections:
>
> <<
> TCP DAOHPRW2:1682 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
> TCP DAOHPRW2:4240 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
> TCP DAOHPRW2:4241 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
> .....
>
> and another 35 established connection the opposite direction:
>
> <<
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:1682 ESTABLISHED
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:4240 ESTABLISHED
> TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:4241 ESTABLISHED
> ....
>
> So, are the connections managed by mod_jk between Apache2 and Tomcat not
> released from the connection pool, despite the connection_pool_timeout=600?
By default apache will use as max number of connections the same value,
as the max number of threads, so that each thread can get it's own
connection. By default it will shrink the connection pool down to half
of the max size. There is a min value you can configure if this doesn't
fit your needs.
> Ideally I'd like to set an upper limit of lets say 200 concurrent HTTP
> Sessions for Tomcat, but I am not sure how exactly to configure the various
> timeout values and limits for mod_jk and Apache accordingly. Our Tomcat
> service makes MySQL queries and it can take at times a few seconds for
> generating a response. Does Apache2 use just one child process handling all
> requests or connections?
Yes, the Windows MPM has one observer process and a second one, that
handles all the requests. If you are really low on server ressources,
you can shrink you connection pool from 250 (equals number of threads
you configured) to 200 and set the minimum to something like 10. You
should also consider a connect_timeout (e.g. 15000) and a
prepost_timeout (e.g. 10000).
Let us know, if your crashes/hangs still show up with the new setup.
Regards,
Rainer
>
> J.Neuhoff
>
>
>
>
> P.S.
> At the moment our server is configured like that:
>
> httpd.conf:
>
> <<
> ....
> Timeout 300
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 15
> # WinNT MPM
> # ThreadsPerChild: constant number of worker threads in the server process
> # MaxRequestsPerChild: maximum number of requests a server process serves
> <IfModule mpm_winnt.c>
> ThreadsPerChild 250
> MaxRequestsPerChild 0
> </IfModule>
> ....
> ### mod_jk 1.x configuration for connecting to Tomcat 5.5
> LoadModule jk_module modules/mod_jk.so
> JkWorkersFile "C:/Program Files/Apache Software Foundation/Tomcat
> 5.5/conf/workers.properties"
> JkLogFile "C:/Program Files/Apache Software Foundation/Tomcat
> 5.5/logs/mod_jk.log"
> JkLogLevel info
> JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
> JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
> JkRequestLogFormat "%w %V %T %r %s"
> JkMount /jsp-examples/* ajp13
> JkMount /servletsp-examples/* ajp13
> JkMount /manager/* ajp13
> JkMount /ohpr/* ajp13
> JkMount /demo-b/* ajp13
> ....
>
> workers.properties:
>
> <<
> # Define 1 real worker using ajp13
> worker.list=ajp13
> # Set properties for worker1 (ajp13)
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
> worker.ajp13.connection_pool_timeout=600
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by JNeuhoff <ne...@mhccorp.com>.
Yesterday, I indeed upgraded another server box to Apache 2.0.59, mod_jk
1.2.20, Tomcat 5.5.17, and also set the connection_pool_timeout=600 (is that
value 600 seconds, or milliseconds?). I then subjected this server to a
brief stresstest (roughly 50 simultanious HTTP sessions on Tomcat's end) and
noticed the following:
Of the two 'Apache.exe' processes running under the Apache2 NT service, one
grew in memory usage quickly from an initial 12MB to nearly 65MB, and stayed
at that level overnight, even though there were no further requests made to
that server since yesterday. I also noticed that a simple 'netstat -a'
reported 35 established connections:
<<
TCP DAOHPRW2:1682 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
TCP DAOHPRW2:4240 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
TCP DAOHPRW2:4241 DAOHPRW2.essex.ac.uk:8009 ESTABLISHED
.....
>>
and another 35 established connection the opposite direction:
<<
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:1682 ESTABLISHED
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:4240 ESTABLISHED
TCP DAOHPRW2:8009 DAOHPRW2.essex.ac.uk:4241 ESTABLISHED
....
>>
So, are the connections managed by mod_jk between Apache2 and Tomcat not
released from the connection pool, despite the connection_pool_timeout=600?
Ideally I'd like to set an upper limit of lets say 200 concurrent HTTP
Sessions for Tomcat, but I am not sure how exactly to configure the various
timeout values and limits for mod_jk and Apache accordingly. Our Tomcat
service makes MySQL queries and it can take at times a few seconds for
generating a response. Does Apache2 use just one child process handling all
requests or connections?
J.Neuhoff
P.S.
At the moment our server is configured like that:
httpd.conf:
<<
....
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_winnt.c>
ThreadsPerChild 250
MaxRequestsPerChild 0
</IfModule>
....
### mod_jk 1.x configuration for connecting to Tomcat 5.5
LoadModule jk_module modules/mod_jk.so
JkWorkersFile "C:/Program Files/Apache Software Foundation/Tomcat
5.5/conf/workers.properties"
JkLogFile "C:/Program Files/Apache Software Foundation/Tomcat
5.5/logs/mod_jk.log"
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T %r %s"
JkMount /jsp-examples/* ajp13
JkMount /servletsp-examples/* ajp13
JkMount /manager/* ajp13
JkMount /ohpr/* ajp13
JkMount /demo-b/* ajp13
....
>>
workers.properties:
<<
# Define 1 real worker using ajp13
worker.list=ajp13
# Set properties for worker1 (ajp13)
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.connection_pool_timeout=600
>>
--
View this message in context: http://www.nabble.com/Apache-mod_jk-memory-leak--tf3023318.html#a8407489
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache mod_jk memory leak?
Posted by Rainer Jung <ra...@kippdata.de>.
Hi,
first 1.2.10 is very ancient. You should upgrade to 1.2.20.
To free (close) connections between apache and tomcat you can then use
the attribute connection_pool_timeout (see documentation of 1.2.20). To
make the connections more robust, you can also use connect_timeout and
prepost_timeout.
Errno 54 on most platforms is "Connection reset by peer", although I'm
not sure about Win 2K3. I don't know exactly how 1.2.10 behaves, but
maybe this was because the remote (=Tomcat) side of the connection had
an idle timeout, but mod_jk not. If that was the reason, this message
should not have a noticable consequence to the users.
There is no memory "leak" known for mod_jk. Most people secure Apache
against leaks by setting MaxRequestsPerChild to an appropriate value.
If you've still got problems after upgrading, you would need to post
relevant parts of your workers.properties, Jk* from apache config and
the connector config.
Note, that info messages really are info, warns or errors are the ones
to look out for (I saw that you had errors for 1.2.10). The info
messages then might help to explain, why warns or errors happened.
Regards,
Rainer
JNeuhoff wrote:
> We are running an Apache 2.0.54 , mod_jk 1.2.10 and Tomcat 5.5.17 on a
> Windows 2003 server box, average web traffic to Tomcat is about 10 to 20
> concurrent HTTP sessions (idle session timeout 15 minutes). After a weekend
> of sudden heavy web traffic with up to 150 simultaneous HTTP sessions we
> experienced 2 server crashes within 24 hours.
>
>
> Even though our Tomcat only ever used 20% of its assigned maximum 512MB JRE
> Memory and never crashed, mod_jk produced loads of error messages along the
> lines:
>
> <<<
> [Sun Jan 14 21:25:37 2007] [info] jk_handler::mod_jk.c (1975): Service
> error=0 for worker=ajp13
> [Sun Jan 14 21:25:40 2007] [error]
> ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
> returned -54 with errno=54
> [Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
> (1220): Error sending request on a fresh connection
> [Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
> Sending request to tomcat failed, recoverable operation attempt=1
> [Sun Jan 14 21:25:40 2007] [error]
> ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
> returned -54 with errno=54
> [Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
> (1220): Error sending request on a fresh connection
> [Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
> Sending request to tomcat failed, recoverable operation attempt=2
> [Sun Jan 14 21:25:40 2007] [error]
> ajp_connection_tcp_send_message::jk_ajp_common.c (918): sendfull
> returned -54 with errno=54
> [Sun Jan 14 21:25:40 2007] [info] ajp_send_request::jk_ajp_common.c
> (1220): Error sending request on a fresh connection
> [Sun Jan 14 21:25:40 2007] [info] ajp_service::jk_ajp_common.c (1724):
> Sending request to tomcat failed, recoverable operation attempt=3
> [Sun Jan 14 21:25:40 2007] [error] ajp_service::jk_ajp_common.c (1733):
> Error connecting to tomcat. Tomcat is probably not started or is
> listening on the wrong port. worker=ajp13 failed
>
> I also noticed that one of the 2 Apache.exe processes went up in memory
> usage from 12MB to a 160MB before Apache failed to serve any requests, even
> though both Apche and Tomcat still continued to log incoming requests.
> It seems that mod_jk doesn't release the memory for unused TCP-connection
> sockets involving port 8009 (the one for the ajp13 protocol). Not even a
> restart of the Apache and Tomcat5 NT services cleared up the pipe, they
> still refused to serve requests. So I finally ended up rebooting the whole
> Windows 2003 server box.
>
> Is there a memory leak problem with Apache 2 / mod_jk ? Any help
> appreciated! How can I configure the system to cope with heavier traffic
> and, if necessary, tell the user that the system is too busy and to come
> back later?
>
> In case it's of any importance, here is the relevant workers.properties for
> the ajp13 we are using:
>
> <<<
> # Set properties for worker1 (ajp13)
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org