You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Pierre Ayotte <pi...@gmail.com> on 2012/06/21 00:26:43 UTC
Failed allocating AJP message buffer
Hello everyone,
We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time we
see a bunch of error "Failed allocating AJP message buffer" and then one
message "All tomcat instances failed, no more workers left" (see below).
The HTTPD.EXE recylcle itself, users are disconnect and then the service
come back online. I did lot of search on those message but i cannot find
any solution. Some help would really be appreciated! See below for more
details
Configuration summary: Apache 2 running mod_ssl + mod_jk load balancing 5
tomcat6 instance.
The mod_jk.log
==============
(.... lots of this AJP message buffer error ....)
[Wed Jun 20 14:59:01.688 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.735 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.750 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.766 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.891 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2393): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.922 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2393): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.938 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.673 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.688 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.719 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.829 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.844 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.860 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.876 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.876 2012] [18324:12832] [error]
service::jk_lb_worker.c (1177): Failed allocating AJP message buffer of
8192 bytes.
[Wed Jun 20 14:59:08.891 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.891 2012] [18324:9768] [error] service::jk_lb_worker.c
(1554): All tomcat instances failed, no more workers left
We also see this error in Windows Event Log:
==============================
====
Faulting application name: httpd.exe, version: 2.2.22.0, time stamp:
0x4f242d7a
Faulting module name: libapr-1.dll, version: 1.4.5.0, time stamp: 0x4f242c33
Exception code: 0xc0000005
Fault offset: 0x00004bd8
Faulting process id: 0x6050
Faulting application start time: 0x01cd4ed4e838b9b3
Faulting application path: D:\Program Files (x86)\Apache Software
Foundation\Apache2.2\bin\httpd.exe
Faulting module path: D:\Program Files (x86)\Apache Software
Foundation\Apache2.2\bin\libapr-1.dll
Report Id: c450d6d3-bb0d-11e1-905f-005056b40061
Our Configuration
=============
Apache
Apache/2.2.22 on Windows 2008 R2 x64
mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in place
mod_jk/1.2.35 with a load balancer to 5 workers
Worker.Properties
worker.list=loadbalancer,status,manager1,manager2,manager3,manager4,manager5
worker.V6_SMInstance1.port=8109
worker.V6_SMInstance1.host=localhost
worker.V6_SMInstance1.type=ajp13
worker.V6_SMInstance1.lbfactor=1
worker.V6_SMInstance2.port=8209
worker.V6_SMInstance2.host=localhost
worker.V6_SMInstance2.type=ajp13
worker.V6_SMInstance2.lbfactor=1
worker.V6_SMInstance3.port=8309
worker.V6_SMInstance3.host=localhost
worker.V6_SMInstance3.type=ajp13
worker.V6_SMInstance3.lbfactor=1
worker.V6_SMInstance4.port=8409
worker.V6_SMInstance4.host=localhost
worker.V6_SMInstance4.type=ajp13
worker.V6_SMInstance4.lbfactor=1
worker.V6_SMInstance5.port=8509
worker.V6_SMInstance5.host=localhost
worker.V6_SMInstance5.type=ajp13
worker.V6_SMInstance5.lbfactor=1
# Status worker for managing load balancer
worker.status.type=status
worker.loadbalancer.type=lb
worker.loadbalancer.sticky_session=1
worker.loadbalancer.balance_workers=V6_SMInstance1,V6_SMInstance2,V6_SMInstance3,V6_SMInstance4,V6_SMInstance5
worker.manager1.port=8109
worker.manager1.host=localhost
worker.manager1.type=ajp13
worker.manager2.port=8209
worker.manager2.host=localhost
worker.manager2.type=ajp13
worker.manager3.port=8309
worker.manager3.host=localhost
worker.manager3.type=ajp13
worker.manager4.port=8409
worker.manager4.host=localhost
worker.manager4.type=ajp13
worker.manager5.port=8509
worker.manager5.host=localhost
worker.manager5.type=ajp13
Tomcat
=============
Tomcat 6.0.32
5 instance of our HP Service Manager application
Regards,
Pierre Ayotte
Re: Failed allocating AJP message buffer
Posted by Pierre Ayotte <pi...@gmail.com>.
Hello Rainer
I already check for memory and for sure it is not a physical memory issu.
We have 12Go and the memory usage is around 6Go at the moment of the crash.
Pierre
2012/6/21 Rainer Jung <ra...@kippdata.de>
> On 21.06.2012 00:26, Pierre Ayotte wrote:
>
>> Hello everyone,
>>
>> We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time
>> we
>> see a bunch of error "Failed allocating AJP message buffer" and then one
>> message "All tomcat instances failed, no more workers left" (see below).
>> The HTTPD.EXE recylcle itself, users are disconnect and then the service
>> come back online. I did lot of search on those message but i cannot find
>> any solution. Some help would really be appreciated!
>>
>
> It seems you are out of memory. Either
>
> - there's some leak in your Apache, most probably in some non standard
> module
>
> - or your server is very tight on memory
>
> - or your application inside Tomcat is getting slow, so there are more and
> more requests getting stuck inside Apache waiting for a response from the
> web app and those requests consume to much memory.
>
> There are MPM settings for Apache on Windows that limit the number of
> threads (=connections =~ concurrent requests) Apache will allow.
>
> Monitor your server memory usage.
>
> Regards,
>
> Rainer
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.org<us...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Failed allocating AJP message buffer
Posted by Rainer Jung <ra...@kippdata.de>.
On 21.06.2012 00:26, Pierre Ayotte wrote:
> Hello everyone,
>
> We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time we
> see a bunch of error "Failed allocating AJP message buffer" and then one
> message "All tomcat instances failed, no more workers left" (see below).
> The HTTPD.EXE recylcle itself, users are disconnect and then the service
> come back online. I did lot of search on those message but i cannot find
> any solution. Some help would really be appreciated!
It seems you are out of memory. Either
- there's some leak in your Apache, most probably in some non standard
module
- or your server is very tight on memory
- or your application inside Tomcat is getting slow, so there are more
and more requests getting stuck inside Apache waiting for a response
from the web app and those requests consume to much memory.
There are MPM settings for Apache on Windows that limit the number of
threads (=connections =~ concurrent requests) Apache will allow.
Monitor your server memory usage.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Failed allocating AJP message buffer
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Pierre,
On 6/27/12 11:36 AM, Pierre Ayotte wrote:
> Just to let you know that installing the latest release of mod_jk
> resolved our issue with http crashing with errors "Failed
> allocating AJP message buffer" in mod_jk.log. So we pass from
> 1.2.35 to 1.2.37.
Good to know. There were a lot of changes to the win32 code around
1.2.35 and some of them caused all sorts of problems. I believe most
of them have been worked out and 1.2.37 should be much more stable for
you.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/rTD4ACgkQ9CaO5/Lv0PAF3gCeJKopRyticr0nah2OQ91svdrh
erwAnAoiFbupvuWaDLDJMw9kAjQAM9nU
=QBCI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Failed allocating AJP message buffer
Posted by Pierre Ayotte <pi...@gmail.com>.
Hi everyone,
Just to let you know that installing the latest release of mod_jk resolved
our issue with http crashing with errors "Failed allocating AJP message
buffer" in mod_jk.log. So we pass from 1.2.35 to 1.2.37.
Regards,
2012/6/22 Pierre Ayotte <pi...@gmail.com>
> Hello Christopher,
>
> Yes we saw on wednesday taht the 1.2.37 has been release sot we just put
> the it in production yesterday night. All keep you inform.
>
> We also make thoses changes to httpd.conf base on recommendation from a
> collegue:
>
> - ThreadsPerChild down to 1000 instead of 3000
> - Remove ThreadLimit 9000 because apparently this does not serve on Windows
>
> Do you think this could help?
>
>
> 2012/6/21 Christopher Schultz <ch...@christopherschultz.net>
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Pierre,
>>
>> On 6/20/12 6:26 PM, Pierre Ayotte wrote:
>> > Our Configuration ============= Apache Apache/2.2.22 on Windows
>> > 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
>> > place mod_jk/1.2.35 with a load balancer to 5 workers
>>
>> There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
>> possible to test with the latest version?
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
>> PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
>> =chWd
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: Failed allocating AJP message buffer
Posted by Pierre Ayotte <pi...@gmail.com>.
Hello Christopher,
Yes we saw on wednesday taht the 1.2.37 has been release sot we just put
the it in production yesterday night. All keep you inform.
We also make thoses changes to httpd.conf base on recommendation from a
collegue:
- ThreadsPerChild down to 1000 instead of 3000
- Remove ThreadLimit 9000 because apparently this does not serve on Windows
Do you think this could help?
2012/6/21 Christopher Schultz <ch...@christopherschultz.net>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Pierre,
>
> On 6/20/12 6:26 PM, Pierre Ayotte wrote:
> > Our Configuration ============= Apache Apache/2.2.22 on Windows
> > 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
> > place mod_jk/1.2.35 with a load balancer to 5 workers
>
> There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
> possible to test with the latest version?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
> PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
> =chWd
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Failed allocating AJP message buffer
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Pierre,
On 6/20/12 6:26 PM, Pierre Ayotte wrote:
> Our Configuration ============= Apache Apache/2.2.22 on Windows
> 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
> place mod_jk/1.2.35 with a load balancer to 5 workers
There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
possible to test with the latest version?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
=chWd
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org