You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Damian Traverso <da...@vostu.com> on 2009/11/01 15:58:44 UTC
Re: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does
not understand this error code: proxy: read response failed
Hi, first of all, thanks for your answer.
And about the error, I also thought that could be the problem, but the
thing is when that error is logged on Apache, the Tomcat instances are
NOT using their MaxThreads capabilities.
Another issue we have is that, when this problem appears, the
processing threads on Tomcat start taking longer responses than
regular usage.
Thanks again,
-Damian
On Sat, Oct 31, 2009 at 7:46 PM, André Warnier <aw...@ice-sa.com> wrote:
> Damian Traverso wrote:
>>
>> Hi,
>> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
>> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
>> are experiencing some issues with high CPU load on the Tomcat side,
>> and Apache starts logging errors like these
>>
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive
>> failed
>> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
>> this error code: proxy: read response failed from
>>
>> On Tomcat side we are using the APR Native Connector, with this settings:
>>
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>> maxThreads="130" backlog="10" connectionTimeout="30000"
>> keepAliveTimeout="30000" enableLookups="false" />
>>
>> Apache MPM conf:
>>
>> ServerLimit 100
>> <IfModule mpm_worker_module>
>> StartServers 30
>> MaxClients 2500
>> MinSpareThreads 50
>> MaxSpareThreads 100
>> ThreadsPerChild 25
>> MaxRequestsPerChild 0
>> </IfModule>
>>
>> mod_proxy_ajp conf:
>>
>> <Proxy balancer://jogacraque>
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2
>> </Proxy>
>>
>> Apache specs:
>>
>> Server version: Apache/2.2.13 (Unix)
>> Server built: Sep 29 2009 11:12:35
>> Server's Module Magic Number: 20051115:23
>> Server loaded: APR 1.3.8, APR-Util 1.3.9
>> Compiled using: APR 1.3.8, APR-Util 1.3.9
>> Architecture: 32-bit
>> Server MPM: Worker
>> threaded: yes (fixed thread count)
>> forked: yes (variable process count)
>>
>>
>> I've beeng testing different settings for mod_proxy_ajp, but I am not
>> sure what is the exact relation between the max option for mod_proxy
>> and ThreadsPerChild. Should I set max to 1? I understand I would have
>> as many connections to each Tomcat box as
>> max*(MaxClients/ThreadsPerChild), is this correct?
>>
>> BTW, is the total number of connections that mod_proxy_ajp will
>> forward to Tomcat related with MaxThreads ? I don't fully understand
>> how APR connector works in this case.
>>
> Hi.
> No great specialist here, but I'll go by the numbers above and give you my
> interpretation.
>
> MaxClients 2500
> on your webserver/balancer indicates how many requests you are willing to
> accept and process at the same time, at the Apache level.
> ThreadsPerChild 25
> means that, if you ever reach these 2500 simultaneous requests, Apache will
> create 100 children processes to handle them, since each child is limited to
> 25 threads, and 2500 / 25 threads -> 100 children.
> (Which seems to match the ServerLimit. Good.)
>
> It is so with the mod_jk connector, and probably also with mod_proy_ajp,
> that the connector will automatically calculate how many connections to make
> to Tomcat, based on the number of threads/processes above.
>
> At the Tomcat level, the AJP Connector says
> maxThreads="130"
> and you have 3 Tomcats, thus for a total maximum number of threads of 130 X
> 3 = 390.
> One Tomcat thread processes one request at a time.
> So the Tomcats, together, can process 390 requests at a time.
>
> There seems to be a mismatch between the number of requests you accept in
> Apache (2500), and the number of threads available to process them in Tomcat
> (390). At the limit, you could have 390 requests being processed in Tomcat,
> and 2110 waiting to get a free Tomcat thread.
>
> That is probably what you are seeing with this log message :
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>
> In other words, probably Apache/mod_proy_ajp got tired of waiting.
>
> On the other hand, you indicate that the poor Tomcats are already struggling
> to cope with the load. Increasing their number of threads is thus unlikely
> to help much.
> So it looks like you need more Tomcats.
> Or else reduce the MaxClients in Apache.
> Or optimise your Tomcat webapps.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this error code: proxy: read response failed
Posted by Ryan Murray <ry...@elementn.com>.
-----Original Message-----
From: Damian Traverso [mailto:damian.traverso@vostu.com]
Sent: Sunday, November 01, 2009 4:59 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] mod_proxy_ajp (2.2.13) + Tomcat 6.0.20 - APR does not understand this error code: proxy: read response failed
Hi, first of all, thanks for your answer.
And about the error, I also thought that could be the problem, but the
thing is when that error is logged on Apache, the Tomcat instances are
NOT using their MaxThreads capabilities.
Another issue we have is that, when this problem appears, the
processing threads on Tomcat start taking longer responses than
regular usage.
Thanks again,
-Damian
On Sat, Oct 31, 2009 at 7:46 PM, André Warnier <aw...@ice-sa.com> wrote:
> Damian Traverso wrote:
>>
>> Hi,
>> we have one Apache-2.2.13 running mod_proxy_ajp + mod_proxy_balancer,
>> connected to (3) Tomcat-6.0.20 instances under Fedora release 8. We
>> are experiencing some issues with high CPU load on the Tomcat side,
>> and Apache starts logging errors like these
>>
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>> [Fri Oct 30 14:47:43 2009] [error] ajp_read_header: ajp_ilink_receive
>> failed
>> [Fri Oct 30 14:47:43 2009] [error] (120006)APR does not understand
>> this error code: proxy: read response failed from
>>
>> On Tomcat side we are using the APR Native Connector, with this settings:
>>
>> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
>> maxThreads="130" backlog="10" connectionTimeout="30000"
>> keepAliveTimeout="30000" enableLookups="false" />
>>
>> Apache MPM conf:
>>
>> ServerLimit 100
>> <IfModule mpm_worker_module>
>> StartServers 30
>> MaxClients 2500
>> MinSpareThreads 50
>> MaxSpareThreads 100
>> ThreadsPerChild 25
>> MaxRequestsPerChild 0
>> </IfModule>
>>
>> mod_proxy_ajp conf:
>>
>> <Proxy balancer://jogacraque>
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app01 ttl=30 max=2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app02 ttl=30 max=2
>> BalancerMember ajp://xxx.xxx.xxx.xxx:8009 route=app03 ttl=30 max=2
>> </Proxy>
>>
>> Apache specs:
>>
>> Server version: Apache/2.2.13 (Unix)
>> Server built: Sep 29 2009 11:12:35
>> Server's Module Magic Number: 20051115:23
>> Server loaded: APR 1.3.8, APR-Util 1.3.9
>> Compiled using: APR 1.3.8, APR-Util 1.3.9
>> Architecture: 32-bit
>> Server MPM: Worker
>> threaded: yes (fixed thread count)
>> forked: yes (variable process count)
>>
>>
>> I've beeng testing different settings for mod_proxy_ajp, but I am not
>> sure what is the exact relation between the max option for mod_proxy
>> and ThreadsPerChild. Should I set max to 1? I understand I would have
>> as many connections to each Tomcat box as
>> max*(MaxClients/ThreadsPerChild), is this correct?
>>
>> BTW, is the total number of connections that mod_proxy_ajp will
>> forward to Tomcat related with MaxThreads ? I don't fully understand
>> how APR connector works in this case.
>>
> Hi.
> No great specialist here, but I'll go by the numbers above and give you my
> interpretation.
>
> MaxClients 2500
> on your webserver/balancer indicates how many requests you are willing to
> accept and process at the same time, at the Apache level.
> ThreadsPerChild 25
> means that, if you ever reach these 2500 simultaneous requests, Apache will
> create 100 children processes to handle them, since each child is limited to
> 25 threads, and 2500 / 25 threads -> 100 children.
> (Which seems to match the ServerLimit. Good.)
>
> It is so with the mod_jk connector, and probably also with mod_proy_ajp,
> that the connector will automatically calculate how many connections to make
> to Tomcat, based on the number of threads/processes above.
>
> At the Tomcat level, the AJP Connector says
> maxThreads="130"
> and you have 3 Tomcats, thus for a total maximum number of threads of 130 X
> 3 = 390.
> One Tomcat thread processes one request at a time.
> So the Tomcats, together, can process 390 requests at a time.
>
> There seems to be a mismatch between the number of requests you accept in
> Apache (2500), and the number of threads available to process them in Tomcat
> (390). At the limit, you could have 390 requests being processed in Tomcat,
> and 2110 waiting to get a free Tomcat thread.
>
> That is probably what you are seeing with this log message :
>> [Fri Oct 30 14:47:43 2009] [error] (70007)The timeout specified has
>> expired: ajp_ilink_receive() can't receive header
>
> In other words, probably Apache/mod_proy_ajp got tired of waiting.
>
> On the other hand, you indicate that the poor Tomcats are already struggling
> to cope with the load. Increasing their number of threads is thus unlikely
> to help much.
> So it looks like you need more Tomcats.
> Or else reduce the MaxClients in Apache.
> Or optimise your Tomcat webapps.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Hi Damian,
I've have experienced these errors for years over many versions of tomcat and APR/mod_ajp, always without a full load on Tomcat. I suspect it's a native socket management issue in the APR lib. I have noticed more errors on some OSs than others. But I'm no expert either.
What I do recommend is to stop using AJP and use mod_proxy_http instead. Unless you need the high degree of tunability of the AJP connectors or the SSL pass through features, IMHO HTTP proxying will give you less headaches and great performance under most loads. I used to have frequent errors, delays and even hanging servers with AJP which I could never diagnose effectively, but have no problems using HTTP proxying.
Also, if you apaches are serving your static content, then configuring more workers at the apache level than Tomcat threads is normal. In fact, if you are serving all your static content from Apache, depending on your hardware, then you may already have too _many_ threads configured on Tomcat. If there are some longer running requests to process which use the app server CPU intensively, you could easily swamp you Tomcat trying to process 130 simultaneous requests. If you've already load tested and this is the right setup for you workload then great.
Cheers,
Ryan
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org