You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Neil Aggarwal <ne...@JAMMConsulting.com> on 2010/06/18 07:19:14 UTC
Application stops responding, jk worker in error state
Hello:
I have Tomcat 6.0.26 running behind Apache and am using the
JK Connector to communicate between them.
My application stops responding on occasion.
I added the jk status page to the web server and checked it
when the application becomes non-responsive.
The tomcat worker has this info:
Worker Status for tomcat
Type Hostname Address:Port Connection Pool Timeout Connect Timeout Prepost
Timeout Reply Timeout Retries Recovery Options Max Packet Size
ajp13 localhost 127.0.0.1:8009 600 0 0 0 2 0
State Acc Err CE RE Wr Rd Busy Max LR LE
ERR 6 (0/sec) 2 0 0 4.8K (7 /sec) 12K (18 /sec) 0 2 663 Fri, 18 Jun 2010
00:06:17 CDT
Note the state is ERR.
Does this mean the worker crashed or does it mean my application is
locked up?
Any ideas how to diagnose this?
Thanks,
Neil
--
Neil Aggarwal, (281)846-8957
FREE trial: Virtualmin VPS with unmetered bandwidth
http://UnmeteredVPS.net/virtualmin
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Posted by Neil Aggarwal <ne...@JAMMConsulting.com>.
Rainer:
> I am attaching the properties for my jk configuation and a
> screen shot of the JK Status in case it helps.
It looks like the mailing list software stripped the
attachments from my email.
Here is the list of jk properties:
worker.server_name=tweb1.retcgroup.com
worker.server_port=80
worker.time_datetime=20100618152329
worker.time_tz=CDT
worker.time_unix=1276892609
worker.web_server=Apache/2.2.3 (CentOS)
worker.jk_version=mod_jk/1.2.30
worker.ajp_count=1
worker.tomcat.list=tomcat
worker.tomcat.type=ajp13
worker.tomcat.host=localhost
worker.tomcat.port=8009
worker.tomcat.address=127.0.0.1:8009
worker.tomcat.connection_pool_timeout=600
worker.tomcat.ping_timeout=10000
worker.tomcat.connect_timeout=0
worker.tomcat.prepost_timeout=0
worker.tomcat.reply_timeout=0
worker.tomcat.retries=2
worker.tomcat.connection_ping_interval=0
worker.tomcat.recovery_options=0
worker.tomcat.max_packet_size=8192
worker.tomcat.used=6043
worker.tomcat.errors=57
worker.tomcat.client_errors=36
worker.tomcat.reply_timeouts=0
worker.tomcat.transferred=1571835650
worker.tomcat.read=65081493100
worker.tomcat.busy=2
worker.tomcat.max_busy=21
worker.tomcat.connected=-39
worker.tomcat.map_count=2
worker.tomcat.last_reset_at=1276866423
worker.tomcat.last_reset_ago=26186
worker.tomcat.error_time_datetime=20100618151913
worker.tomcat.error_time_tz=CDT
worker.tomcat.error_time_unix seconds=1276892353
worker.tomcat.error_time_ago seconds=256
worker.tomcat.map.1.server=tweb1.retcgroup.com
worker.tomcat.map.1.uri=/thymeleweb/*
worker.tomcat.map.1.type=Wildchar
worker.tomcat.map.1.source=JkMount
worker.tomcat.map.1.reply_timeout=-1
worker.tomcat.map.1.fail_on_status=
worker.tomcat.map.1.active=
worker.tomcat.map.1.disabled=
worker.tomcat.map.1.stopped=
worker.tomcat.map.1.use_server_errors=0
worker.tomcat.map.2.server=tweb1.retcgroup.com [_default_:443]
worker.tomcat.map.2.uri=/thymeleweb/*
worker.tomcat.map.2.type=Wildchar
worker.tomcat.map.2.source=JkMount
worker.tomcat.map.2.reply_timeout=-1
worker.tomcat.map.2.fail_on_status=
worker.tomcat.map.2.active=
worker.tomcat.map.2.disabled=
worker.tomcat.map.2.stopped=
worker.tomcat.map.2.use_server_errors=0
worker.result.type=OK
worker.result.message=Action finished
The screenshot of my jk status page is at:
http://tweb1.retcgroup.com/screenshot.gif
Thanks,
Neil
--
Neil Aggarwal, (281)846-8957, http://UnmeteredVPS.net
FREE trial: Tomcat 6 VPS with unmetered bandwidth
http://UnmeteredVPS.net/tomcat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Posted by Neil Aggarwal <ne...@JAMMConsulting.com>.
Rainer:
> would you mind opening an issua in Bugzilla, attaching your
> log snippet
> and possibly your workers.properties
I created bug 49468 in bugzilla, included my log snippet and
attached the workers.properties file.
> You can also open a second issue concering the obvisouly wrong
> connection count shown in the status worker.
I created bug 49469 in bugzilla and attached my screenshot
showing the negative number of connections.
> Thanks and sory for the trouble
Thank you for being so responsive!
I hope we are able to fix the lock-up problem soon since this
is a production system. I sincerely appreciate your help.
Thanks,
Neil
--
Neil Aggarwal, (281)846-8957, http://UnmeteredVPS.net
FREE trial: Tomcat 6 virtual server with unmetered bandwidth
http://UnmeteredVPS.net/tomcat
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [SPAM] RE: Application stops responding, jk worker in error state
Posted by Rainer Jung <ra...@kippdata.de>.
On 19.06.2010 03:46, Neil Aggarwal wrote:
> Rainer:
>
>> Hmmm, errno 11 is often EAGAIN. What platform are you using
>> (OS, version)?
>
> I am using CentOS 5.5 freshly installed, not an upgrade.
> I tried that as a fix to this problem. I wiped the server
> clean and reinstalled everything from scratch. It was on
> CentOS 5.4 before and I had the same behavior.
>
>>> What is really strange is the JK Status page says -48
>> connections. It seems
>>> like that number should always be positive or zero but not negative.
>>
>> That should be unrelated (and yes: it's not OK).
>
> It seems to me if jk is confused about the number of
> backend connections, it may not be connecting correctly
> to Tomcat. That seems like it would cause the behavior
> I am experiencing.
No, the total number of connections is shown only for convenience
(monitoring etc.). The real process pool is local for each Apache
process. there seems to be something wrong when tracking the total
number via shared memory but that shouldn't influence in any way the
usual functionality. It has only been added very recently.
What *is* a problem is EAGAIN while reading from the socket. We don't
expect that and handle it as an error, though strictly speaking it is
only a temporary error condition. i'll see whether I can provide a patch.
would you mind opening an issua in Bugzilla, attaching your log snippet
and possibly your workers.properties (the property dump from the status
worker is a little harder to digest, because it also shows all the
defaults).
You can also open a second issue concering the obvisouly wrong
connection count shown in the status worker.
Thanks and sory for the trouble
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Posted by Neil Aggarwal <ne...@JAMMConsulting.com>.
Rainer:
> Hmmm, errno 11 is often EAGAIN. What platform are you using
> (OS, version)?
I am using CentOS 5.5 freshly installed, not an upgrade.
I tried that as a fix to this problem. I wiped the server
clean and reinstalled everything from scratch. It was on
CentOS 5.4 before and I had the same behavior.
> > What is really strange is the JK Status page says -48
> connections. It seems
> > like that number should always be positive or zero but not negative.
>
> That should be unrelated (and yes: it's not OK).
It seems to me if jk is confused about the number of
backend connections, it may not be connecting correctly
to Tomcat. That seems like it would cause the behavior
I am experiencing.
Thanks,
Neil
--
Neil Aggarwal, (281)846-8957
FREE trial: Virtualmin VPS with unmetered bandwidth
http://UnmeteredVPS.net/virtualmin
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Application stops responding, jk worker in error state
Posted by Rainer Jung <ra...@kippdata.de>.
On 18.06.2010 22:44, Neil Aggarwal wrote:
> Rainer:
>
>> If you are going to post part of your log file, please also
>> do provide
>> your mod_jk configuration and version information.
>
> I am using mod_jk 1.2.30 which I built from the source tarball.
>
> My application just locked up so I did a tail -f
> on the mod_jk log and made a request to it.
>
> Here is what generated in the log:
>
> [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info]
> ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't
> receive the response header message from tomcat, network problems or tomcat
> (127.0.0.1:8009) is down (errno=11)
Hmmm, errno 11 is often EAGAIN. What platform are you using (OS, version)?
> [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [error]
> ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused
> connection. No response has been sent to the client (yet)
> [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info]
> ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat
> failed (recoverable), (attempt=1)
> [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
> ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't
> receive the response header message from tomcat, network problems or tomcat
> (127.0.0.1:8009) is down (errno=11)
> [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error]
> ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused
> connection. No response has been sent to the client (yet)
> [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
> ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat
> failed (recoverable), (attempt=2)
> [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error]
> ajp_service::jk_ajp_common.c (2559): (tomcat) connecting to tomcat failed.
> [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
> jk_handler::mod_jk.c (2618): Service error=0 for worker=tomcat
>
> I took a look at the tomcat catalina.out and do not see any errors there.
> The CPU is empty and the RAM is available.
>
> I telnet to 127.0.0.1 port 8009 on the machine, I get a connection,
> so something is listening on the port.
>
> If I do a ps aux | grep java, I see two processes for tomcat which is
> normal.
>
> I checked the mysql log and do not see any errors. I did a show innodb
> status and do not see any deadlocks in the database.
>
> I am attaching the properties for my jk configuation and a screen shot of
> the
> JK Status in case it helps.
>
> What is really strange is the JK Status page says -48 connections. It seems
> like that number should always be positive or zero but not negative.
That should be unrelated (and yes: it's not OK).
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Posted by Neil Aggarwal <ne...@JAMMConsulting.com>.
Rainer:
> If you are going to post part of your log file, please also
> do provide
> your mod_jk configuration and version information.
I am using mod_jk 1.2.30 which I built from the source tarball.
My application just locked up so I did a tail -f
on the mod_jk log and made a request to it.
Here is what generated in the log:
[Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info]
ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't
receive the response header message from tomcat, network problems or tomcat
(127.0.0.1:8009) is down (errno=11)
[Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [error]
ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info]
ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat
failed (recoverable), (attempt=1)
[Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't
receive the response header message from tomcat, network problems or tomcat
(127.0.0.1:8009) is down (errno=11)
[Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error]
ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused
connection. No response has been sent to the client (yet)
[Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat
failed (recoverable), (attempt=2)
[Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error]
ajp_service::jk_ajp_common.c (2559): (tomcat) connecting to tomcat failed.
[Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info]
jk_handler::mod_jk.c (2618): Service error=0 for worker=tomcat
I took a look at the tomcat catalina.out and do not see any errors there.
The CPU is empty and the RAM is available.
I telnet to 127.0.0.1 port 8009 on the machine, I get a connection,
so something is listening on the port.
If I do a ps aux | grep java, I see two processes for tomcat which is
normal.
I checked the mysql log and do not see any errors. I did a show innodb
status and do not see any deadlocks in the database.
I am attaching the properties for my jk configuation and a screen shot of
the
JK Status in case it helps.
What is really strange is the JK Status page says -48 connections. It seems
like that number should always be positive or zero but not negative.
I appreciate any help you can offer.
Thanks,
Neil
--
Neil Aggarwal, (281)846-8957
FREE trial: Tomcat 6 VPS with unmetered bandwidth
http://UnmeteredVPS.net/tomcat
Re: Application stops responding, jk worker in error state
Posted by Rainer Jung <ra...@kippdata.de>.
On 18.06.2010 07:19, Neil Aggarwal wrote:
> Hello:
>
> I have Tomcat 6.0.26 running behind Apache and am using the
> JK Connector to communicate between them.
>
> My application stops responding on occasion.
>
> I added the jk status page to the web server and checked it
> when the application becomes non-responsive.
>
> The tomcat worker has this info:
>
> Worker Status for tomcat
> Type Hostname Address:Port Connection Pool Timeout Connect Timeout Prepost
> Timeout Reply Timeout Retries Recovery Options Max Packet Size
> ajp13 localhost 127.0.0.1:8009 600 0 0 0 2 0
>
> State Acc Err CE RE Wr Rd Busy Max LR LE
> ERR 6 (0/sec) 2 0 0 4.8K (7 /sec) 12K (18 /sec) 0 2 663 Fri, 18 Jun 2010
> 00:06:17 CDT
>
> Note the state is ERR.
>
> Does this mean the worker crashed or does it mean my application is
> locked up?
>
> Any ideas how to diagnose this?
It means mod-jk detected a communication error when trying to interact
with Tomcat. Like it couldn't open a connection, it didn't get the
answer, ... The exact types of detected errors depend on your mod_jk
configuration. How to diagnose? Look at the mod_jk error log. Whenever a
worker is put into error state, there will be at least one log message
of type "[error]" in the mod-jk log. If you increase your JkLogLevel to
info, the info log lines around the error line will contain more hints
about the root cause, info level messages without an accompanying error
message are harmless.
If you are going to post part of your log file, please also do provide
your mod_jk configuration and version information.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org