You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Django Radonich-Camp <dj...@emergeinteractive.com> on 2012/09/19 23:38:37 UTC
ajp_ilink_receive error - please advise
hello.
we are running an application on tomcat and experiencing intermittent
periods where the application is non-responsive and thus non-functional.
the general set up is apache and tomcat, with mod_proxy_ajp as the
connector (specific details and configs below).
during these events, the primary apache error log shows the following:
1. (104)Connection reset by peer: ajp_ilink_receive() can't receive header
2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
receive header
it looks like error #1 repeats for a while and then error #2 repeats for a
while.
at the same time in the application specific apache error log (as specified
in the vhost set up for app) we see the following errors repeated (though
in mixed order from below):
3. [error] ajp_read_header: ajp_ilink_receive failed
4. [error] (120006)APR does not understand this error code: proxy: read
response failed from (null) (localhost)
5. [error] (120006)APR does not understand this error code: proxy: read
response failed from 127.0.0.1:8009 (localhost)
6. [error] [client X.X.X.X] proxy: error processing end
and occasionally:
7. [error] proxy: read zero bytes, expecting 464 bytes
the catalina.out log registers nothing during the time period the
application is unresponsive.
a couple of other things to note:
- these events are coming under light to no load as far as i can tell.
- these events last from about 5 to 30 minutes and then everything works
again as expected with no manual intervention.
- the time of day of for the events is not consistent.
- these events initially occured rarely, but over the last month have
ramped up to daily.
any suggestions on how to fix or further troubleshoot this problem? and
thanks in advance for the help.
below please find more information on versions and configs... let me know
if more info is needed.
OS Name: Linux (ubuntu 10.04.4)
OS Version: 2.6.32-31-server
Architecture: amd64
JVM Version: 1.6.0_32-b05
JVM Vendor: Sun Microsystems Inc.
Server version: Apache Tomcat/6.0.24
apache: Apache/2.2.14
MPM configs
--------START--------------
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 26
MaxRequestsPerChild 1000
</IfModule>
--------END----------------
balancer conf
--------START--------------
<Proxy balancer://mysite_balancer*>
Order deny,allow
Allow from all
</Proxy>
ProxyPassMatch ^/(.+.cf[cm])(.*)?$ balancer://mysite_balancer/irised/client
stickysession=JSESSIONID|jsessionid
ProxyPassReverseCookiePath /irised /
<Proxy balancer://mysite_balancer>
BalancerMember ajp://localhost:8009 route=www1 retry=5
</Proxy>
--------END----------------
connector xml
--------START--------------
<Connector URIEncoding="UTF-8"
port="8009"
protocol="AJP/1.3"
connectionTimeout="20000"
redirectPort="8443" />
--------END----------------
Re: ajp_ilink_receive error - please advise
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Django,
On 9/19/12 5:38 PM, Django Radonich-Camp wrote:
> apache: Apache/2.2.14
That's 9 revisions out of date. Any chance you can upgrade to 2.2.23
and re-test? Is it reliably reproducible?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBaTyUACgkQ9CaO5/Lv0PDGRQCdGr9fhj2dyiH4hxysDlyrGMH/
KN0An3tbQN6JTRPQfJnEt0VRvOYz9wSl
=WMXH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ajp_ilink_receive error - please advise
Posted by Django Radonich-Camp <dj...@emergeinteractive.com>.
sorry to take so long to respond, but i was finally able to capture a
thread dump during one of these events...
the thread dump (taken via jstack) is view-able here:
https://gist.github.com/aa772c922feee65177c1
any help deciphering it would be appreciated. no deadlocks, but it does
appear there is blocking. how does one figure out what is blocking what?
or what is the cause of the initial block?
thanks.
On Fri, Sep 21, 2012 at 8:44 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dan,
>
> On 9/21/12 5:00 PM, Daniel Mikusa wrote:
> >
> > There are a few ways to [get a thread dump].
>
>
> http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlBdNDkACgkQ9CaO5/Lv0PC0WACgkd2+QU7MsT5diGB316NSP601
> FZcAn26ek5nGMyhaKTnqr30g9plW/+oo
> =y54N
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Django Radonich-Camp
Technical Lead
Emerge Interactive Inc.
930 NW 14th Ave, Suite 280
Portland, OR 97209
Tel. 503-922-3483
Fax. 503-296-5784
http://www.emergeinteractive.com
Re: ajp_ilink_receive error - please advise
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dan,
On 9/21/12 5:00 PM, Daniel Mikusa wrote:
>
> There are a few ways to [get a thread dump].
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBdNDkACgkQ9CaO5/Lv0PC0WACgkd2+QU7MsT5diGB316NSP601
FZcAn26ek5nGMyhaKTnqr30g9plW/+oo
=y54N
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ajp_ilink_receive error - please advise
Posted by Daniel Mikusa <dm...@vmware.com>.
On Sep 20, 2012, at 6:12 PM, Django Radonich-Camp wrote:
> thank you for the replies chris and dan.
>
> chris:
> 1. upgrading apache is possible, but not easy (puppet manages the install).
> 2. unfortunately i have not figured out how to reproduce the problem. we
> did extensive load testing on the system prior to launch and not once did
> we see this problem...
>
> dan:
> 1. good idea. do you know a good reference for setting this up?
It is just a matter of adding the JVM options in setenv.sh.
I'll typically use something like this...
-XX:+PrintGC -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
You can adjust the options to fit your needs. If you need more detail, I would suggest google'ing "java garbage collection log" or the individual option.
> 2. also, what is the process for capturing thread dumps? this will be
> difficult however because the incidents are random and so far, mostly
> ocurring in the middle of the night.
There are a few ways to do this. I find running "kill -3 <pid>" to be the easiest. You could also use "jstack <pid>"
Dan
>
> thanks again for the help.
>
>
> On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa <dm...@vmware.com> wrote:
>
>> On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:
>>
>>> hello.
>>>
>>> we are running an application on tomcat and experiencing intermittent
>>> periods where the application is non-responsive and thus non-functional.
>>> the general set up is apache and tomcat, with mod_proxy_ajp as the
>>> connector (specific details and configs below).
>>>
>>> during these events, the primary apache error log shows the following:
>>> 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive
>> header
>>> 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
>>> receive header
>>>
>>> it looks like error #1 repeats for a while and then error #2 repeats for
>> a
>>> while.
>>>
>>> at the same time in the application specific apache error log (as
>> specified
>>> in the vhost set up for app) we see the following errors repeated (though
>>> in mixed order from below):
>>> 3. [error] ajp_read_header: ajp_ilink_receive failed
>>> 4. [error] (120006)APR does not understand this error code: proxy: read
>>> response failed from (null) (localhost)
>>> 5. [error] (120006)APR does not understand this error code: proxy: read
>>> response failed from 127.0.0.1:8009 (localhost)
>>> 6. [error] [client X.X.X.X] proxy: error processing end
>>>
>>> and occasionally:
>>> 7. [error] proxy: read zero bytes, expecting 464 bytes
>>>
>>> the catalina.out log registers nothing during the time period the
>>> application is unresponsive.
>>>
>>> a couple of other things to note:
>>> - these events are coming under light to no load as far as i can tell.
>>> - these events last from about 5 to 30 minutes and then everything works
>>> again as expected with no manual intervention.
>>> - the time of day of for the events is not consistent.
>>> - these events initially occured rarely, but over the last month have
>>> ramped up to daily.
>>>
>>> any suggestions on how to fix or further troubleshoot this problem? and
>>> thanks in advance for the help.
>>
>> Here are a couple additional troubleshooting steps you can take on your
>> Tomcat instances...
>>
>> 1.) Enable garbage collection logging. Look for any full GC's.
>> 2.) Take some thread dumps during an incident. Look for blocking.
>>
>> Dan
>>
>>
>>>
>>> below please find more information on versions and configs... let me know
>>> if more info is needed.
>>>
>>> OS Name: Linux (ubuntu 10.04.4)
>>> OS Version: 2.6.32-31-server
>>> Architecture: amd64
>>> JVM Version: 1.6.0_32-b05
>>> JVM Vendor: Sun Microsystems Inc.
>>> Server version: Apache Tomcat/6.0.24
>>> apache: Apache/2.2.14
>>>
>>> MPM configs
>>> --------START--------------
>>> <IfModule mpm_prefork_module>
>>> StartServers 5
>>> MinSpareServers 5
>>> MaxSpareServers 10
>>> MaxClients 26
>>> MaxRequestsPerChild 1000
>>> </IfModule>
>>> --------END----------------
>>>
>>> balancer conf
>>> --------START--------------
>>> <Proxy balancer://mysite_balancer*>
>>> Order deny,allow
>>> Allow from all
>>> </Proxy>
>>>
>>> ProxyPassMatch ^/(.+.cf[cm])(.*)?$
>> balancer://mysite_balancer/irised/client
>>> stickysession=JSESSIONID|jsessionid
>>>
>>> ProxyPassReverseCookiePath /irised /
>>>
>>> <Proxy balancer://mysite_balancer>
>>> BalancerMember ajp://localhost:8009 route=www1 retry=5
>>> </Proxy>
>>> --------END----------------
>>>
>>> connector xml
>>> --------START--------------
>>> <Connector URIEncoding="UTF-8"
>>> port="8009"
>>> protocol="AJP/1.3"
>>> connectionTimeout="20000"
>>> redirectPort="8443" />
>>> --------END----------------
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: ajp_ilink_receive error - please advise
Posted by Django Radonich-Camp <dj...@emergeinteractive.com>.
thank you for the replies chris and dan.
chris:
1. upgrading apache is possible, but not easy (puppet manages the install).
2. unfortunately i have not figured out how to reproduce the problem. we
did extensive load testing on the system prior to launch and not once did
we see this problem...
dan:
1. good idea. do you know a good reference for setting this up?
2. also, what is the process for capturing thread dumps? this will be
difficult however because the incidents are random and so far, mostly
ocurring in the middle of the night.
thanks again for the help.
On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa <dm...@vmware.com> wrote:
> On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:
>
> > hello.
> >
> > we are running an application on tomcat and experiencing intermittent
> > periods where the application is non-responsive and thus non-functional.
> > the general set up is apache and tomcat, with mod_proxy_ajp as the
> > connector (specific details and configs below).
> >
> > during these events, the primary apache error log shows the following:
> > 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive
> header
> > 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
> > receive header
> >
> > it looks like error #1 repeats for a while and then error #2 repeats for
> a
> > while.
> >
> > at the same time in the application specific apache error log (as
> specified
> > in the vhost set up for app) we see the following errors repeated (though
> > in mixed order from below):
> > 3. [error] ajp_read_header: ajp_ilink_receive failed
> > 4. [error] (120006)APR does not understand this error code: proxy: read
> > response failed from (null) (localhost)
> > 5. [error] (120006)APR does not understand this error code: proxy: read
> > response failed from 127.0.0.1:8009 (localhost)
> > 6. [error] [client X.X.X.X] proxy: error processing end
> >
> > and occasionally:
> > 7. [error] proxy: read zero bytes, expecting 464 bytes
> >
> > the catalina.out log registers nothing during the time period the
> > application is unresponsive.
> >
> > a couple of other things to note:
> > - these events are coming under light to no load as far as i can tell.
> > - these events last from about 5 to 30 minutes and then everything works
> > again as expected with no manual intervention.
> > - the time of day of for the events is not consistent.
> > - these events initially occured rarely, but over the last month have
> > ramped up to daily.
> >
> > any suggestions on how to fix or further troubleshoot this problem? and
> > thanks in advance for the help.
>
> Here are a couple additional troubleshooting steps you can take on your
> Tomcat instances...
>
> 1.) Enable garbage collection logging. Look for any full GC's.
> 2.) Take some thread dumps during an incident. Look for blocking.
>
> Dan
>
>
> >
> > below please find more information on versions and configs... let me know
> > if more info is needed.
> >
> > OS Name: Linux (ubuntu 10.04.4)
> > OS Version: 2.6.32-31-server
> > Architecture: amd64
> > JVM Version: 1.6.0_32-b05
> > JVM Vendor: Sun Microsystems Inc.
> > Server version: Apache Tomcat/6.0.24
> > apache: Apache/2.2.14
> >
> > MPM configs
> > --------START--------------
> > <IfModule mpm_prefork_module>
> > StartServers 5
> > MinSpareServers 5
> > MaxSpareServers 10
> > MaxClients 26
> > MaxRequestsPerChild 1000
> > </IfModule>
> > --------END----------------
> >
> > balancer conf
> > --------START--------------
> > <Proxy balancer://mysite_balancer*>
> > Order deny,allow
> > Allow from all
> > </Proxy>
> >
> > ProxyPassMatch ^/(.+.cf[cm])(.*)?$
> balancer://mysite_balancer/irised/client
> > stickysession=JSESSIONID|jsessionid
> >
> > ProxyPassReverseCookiePath /irised /
> >
> > <Proxy balancer://mysite_balancer>
> > BalancerMember ajp://localhost:8009 route=www1 retry=5
> > </Proxy>
> > --------END----------------
> >
> > connector xml
> > --------START--------------
> > <Connector URIEncoding="UTF-8"
> > port="8009"
> > protocol="AJP/1.3"
> > connectionTimeout="20000"
> > redirectPort="8443" />
> > --------END----------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: ajp_ilink_receive error - please advise
Posted by Daniel Mikusa <dm...@vmware.com>.
On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:
> hello.
>
> we are running an application on tomcat and experiencing intermittent
> periods where the application is non-responsive and thus non-functional.
> the general set up is apache and tomcat, with mod_proxy_ajp as the
> connector (specific details and configs below).
>
> during these events, the primary apache error log shows the following:
> 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive header
> 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
> receive header
>
> it looks like error #1 repeats for a while and then error #2 repeats for a
> while.
>
> at the same time in the application specific apache error log (as specified
> in the vhost set up for app) we see the following errors repeated (though
> in mixed order from below):
> 3. [error] ajp_read_header: ajp_ilink_receive failed
> 4. [error] (120006)APR does not understand this error code: proxy: read
> response failed from (null) (localhost)
> 5. [error] (120006)APR does not understand this error code: proxy: read
> response failed from 127.0.0.1:8009 (localhost)
> 6. [error] [client X.X.X.X] proxy: error processing end
>
> and occasionally:
> 7. [error] proxy: read zero bytes, expecting 464 bytes
>
> the catalina.out log registers nothing during the time period the
> application is unresponsive.
>
> a couple of other things to note:
> - these events are coming under light to no load as far as i can tell.
> - these events last from about 5 to 30 minutes and then everything works
> again as expected with no manual intervention.
> - the time of day of for the events is not consistent.
> - these events initially occured rarely, but over the last month have
> ramped up to daily.
>
> any suggestions on how to fix or further troubleshoot this problem? and
> thanks in advance for the help.
Here are a couple additional troubleshooting steps you can take on your Tomcat instances...
1.) Enable garbage collection logging. Look for any full GC's.
2.) Take some thread dumps during an incident. Look for blocking.
Dan
>
> below please find more information on versions and configs... let me know
> if more info is needed.
>
> OS Name: Linux (ubuntu 10.04.4)
> OS Version: 2.6.32-31-server
> Architecture: amd64
> JVM Version: 1.6.0_32-b05
> JVM Vendor: Sun Microsystems Inc.
> Server version: Apache Tomcat/6.0.24
> apache: Apache/2.2.14
>
> MPM configs
> --------START--------------
> <IfModule mpm_prefork_module>
> StartServers 5
> MinSpareServers 5
> MaxSpareServers 10
> MaxClients 26
> MaxRequestsPerChild 1000
> </IfModule>
> --------END----------------
>
> balancer conf
> --------START--------------
> <Proxy balancer://mysite_balancer*>
> Order deny,allow
> Allow from all
> </Proxy>
>
> ProxyPassMatch ^/(.+.cf[cm])(.*)?$ balancer://mysite_balancer/irised/client
> stickysession=JSESSIONID|jsessionid
>
> ProxyPassReverseCookiePath /irised /
>
> <Proxy balancer://mysite_balancer>
> BalancerMember ajp://localhost:8009 route=www1 retry=5
> </Proxy>
> --------END----------------
>
> connector xml
> --------START--------------
> <Connector URIEncoding="UTF-8"
> port="8009"
> protocol="AJP/1.3"
> connectionTimeout="20000"
> redirectPort="8443" />
> --------END----------------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org