You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Vasily Ivanov <ba...@gmail.com> on 2006/05/08 04:45:00 UTC
Tomcat 4.1.13 dies every 3-4 days
Hello,
We are running J2EE web-application under Apache Server/Tomcat 4.1.31 on Linux.
At any point of time we've got around 250 active open sessions and
around 100 active threads running on server.
The problem is that tomcat occasionally gets dead every 3-4 days
without reporting any error to logs. There are 2Gb of memory allocated
to tomcat. And memory usage is normally 5-10%.
Some logs and server.xml attached.
I can not reproduce this in the dev environment.
Logs:
Here is the error from the mod_jk.log:
===================================================
[Mon May 08 10:25:14 2006] worker1 appl.company.com.au 0.004600
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.002706
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.013406
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.008658
[Mon May 08 10:25:15 2006] worker1 appl.company.com.au 0.002049
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 0.335327
[Mon May 08 10:25:16 2006] worker1 appl 0.009130
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 1.030997
[Mon May 08 10:25:16 2006] worker1 appl 0.008127
[Mon May 08 10:25:16 2006] worker1 appl 0.019742
[Mon May 08 10:25:16 2006] worker1 appl 0.007065
[Mon May 08 10:25:16 2006] worker1 appl.company.com.au 4.910941
[Mon May 08 10:25:17 2006] worker1 appl.company.com.au 6.059512
[Mon May 08 10:25:18 2006] [info]
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:18 2006] [info] ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 10.370808
[Mon May 08 10:25:18 2006] [info] jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 2.020834
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 2.062616
[Mon May 08 10:25:18 2006] [info]
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:18 2006] [info] ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:18 2006] worker1 appl.company.com.au 7.267761
[Mon May 08 10:25:18 2006] [info] jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 1.804931
[Mon May 08 10:25:20 2006] [info]
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:20 2006] [info] ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 5.307884
[Mon May 08 10:25:20 2006] [info] jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:20 2006] worker1 appl.company.com.au 7.851162
[Mon May 08 10:25:21 2006] worker1 appl.company.com.au 0.229151
[Mon May 08 10:25:21 2006] [info]
ajp_process_callback::jk_ajp_common.c (1363): Connection aborted or
network problems
[Mon May 08 10:25:21 2006] [info] ajp_service::jk_ajp_common.c
(1706): Receiving from tomcat failed, because of client error without
recovery in send loop 0
[Mon May 08 10:25:21 2006] worker1 appl.company.com.au 13.170979
[Mon May 08 10:25:21 2006] [info] jk_handler::mod_jk.c (1968):
Aborting connection for worker=worker1
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 6.087992
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 0.006206
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 11.215213
[Mon May 08 10:25:25 2006] worker1 appl.company.com.au 1.396763
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error] ajp_get_reply::jk_ajp_common.c
(1479): Tomcat is down or network problems. No response has been sent
to the client (yet)
[Mon May 08 10:25:26 2006] [info] ajp_service::jk_ajp_common.c
(1696): Receiving from tomcat failed, recoverable operation attempt=0
[Mon May 08 10:25:26 2006] [info] ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed, recoverable operation
attempt=1
[Mon May 08 10:25:26 2006] [info] jk_open_socket::jk_connect.c (433):
connect to 127.0.0.1:8009 failed with errno=111
[Mon May 08 10:25:26 2006] [info]
ajp_connect_to_endpoint::jk_ajp_common.c (877): Failed connecting to
tomcat. Tomcat is probably not started or is listening on the wrong
host/port (127.0.0.1:8009). Failed errno = 111
[Mon May 08 10:25:26 2006] [info] ajp_send_request::jk_ajp_common.c
(1227): Error connecting to the Tomcat process.
[Mon May 08 10:25:26 2006] [info] ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed, recoverable operation
attempt=2
[Mon May 08 10:25:26 2006] [info] jk_open_socket::jk_connect.c (433):
connect to 127.0.0.1:8009 failed with errno=111
[Mon May 08 10:25:26 2006] [info]
ajp_connect_to_endpoint::jk_ajp_common.c (877): Failed connecting to
tomcat. Tomcat is probably not started or is listening on the wrong
host/port (127.0.0.1:8009). Failed errno = 111
[Mon May 08 10:25:26 2006] [info] ajp_send_request::jk_ajp_common.c
(1227): Error connecting to the Tomcat process.
[Mon May 08 10:25:26 2006] [info] ajp_service::jk_ajp_common.c
(1724): Sending request to tomcat failed, recoverable operation
attempt=3
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (949): ERROR: can't
receive the response message from tomcat, network problems or tomcat
is down (127.0.0.1:8009), err=-2
[Mon May 08 10:25:26 2006] [error] ajp_get_reply::jk_ajp_common.c
(1479): Tomcat is down or network problems. No response has been sent
to the client (yet)
===================================================
Here is the error_log from around this time:
===================================================
[Mon May 08 10:10:47 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=AE86ED2E4E74E1984F44046CA8D3A1B1,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:21 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=BFF2FDF5D0988AB6912B337DB852B9A0,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:21 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=BFF2FDF5D0988AB6912B337DB852B9A0,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:12:48 2006] [error] [client 161.117.83.199] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=A78A0AC23EB86064B5C9A725D77BE40B,
referer: http://appl/appl/authentication/authenticatestart.do
[Mon May 08 10:12:48 2006] [error] [client 161.117.83.199] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=A78A0AC23EB86064B5C9A725D77BE40B,
referer: http://appl/appl/authentication/authenticatestart.do
[Mon May 08 10:14:42 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=4337CBD952FBB9FC6D8607E9C5FDD02D,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:14:42 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/javascript/common.js;jsessionid=4337CBD952FBB9FC6D8607E9C5FDD02D,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
[Mon May 08 10:14:51 2006] [error] [client 161.117.238.233] File does
not exist: /app/appl/current/web/appl/styles/standard.css;jsessionid=8F8A9C12B013F6BDF00E05EEEA89C42F,
referer: http://appl.company.com.au/appl/authentication/authenticatestart.do
===================================================
Here is the tomcat process after restart:
===================================================
tomcat 9791 1 99 10:44 pts/2 00:01:03
/usr/j2sdk1.4.2_04/bin/java -server -Dappl_1 -Xms2048m -Xmx2048m
-Djava.endorsed.dirs=/app/tomcat/current/common/endorsed -classpath
/usr/j2sdk1.4.2_04/lib/tools.jar:/app/tomcat/current/bin/bootstrap.jar
-Dcatalina.base=/app/appl/tcinst1 -Dcatalina.home=/app/tomcat/curr
===================================================
Here is application log (not full) after restart:
===================================================
2006-05-08 10:44:06,857 [main] Http11Protocol INFO - Initializing
Coyote HTTP/1.1 on http-8029
Starting service Tomcat-Standalone
Apache Tomcat/4.1.31
...
8/05/2006 10:44:34 org.apache.commons.digester.Digester error
SEVERE: Parse Error at line 754 column 16: The content of element type
"definition" must match
"(icon?,display-name?,description?,put*,putList*)".
org.xml.sax.SAXParseException: The content of element type
"definition" must match
"(icon?,display-name?,description?,put*,putList*)".
at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown
Source)
at org.apache.xerces.util.ErrorHandlerWrapper.error(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
at org.apache.xerces.impl.dtd.XMLDTDValidator.handleEndElement(Unknown Source)
at org.apache.xerces.impl.dtd.XMLDTDValidator.endElement(Unknown Source)
...
2006-05-08 10:44:35,672 [main] JkMain INFO - Jk running ID=0
time=17/90 config=/app/appl/tcinst1/conf/jk2.properties
...
2006-05-08 10:46:34,762 [TP-Processor23] Registry ERROR - Error
registering Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest25
javax.management.InstanceAlreadyExistsException:
Catalina:type=RequestProcessor,worker=jk-8009,name=JkRequest25
at mx4j.server.MBeanServerImpl.register(MBeanServerImpl.java:1123)
at mx4j.server.MBeanServerImpl.registerImpl(MBeanServerImpl.java:1054)
at mx4j.server.MBeanServerImpl.registerMBeanImpl(MBeanServerImpl.java:1002)
at mx4j.server.MBeanServerImpl.registerMBean(MBeanServerImpl.java:978)
at org.apache.commons.modeler.Registry.registerComponent(Registry.java:871)
...
==================================================
This is server.xml
===================================================
<Server port="8005" shutdown="SHUTDOWN" debug="0">
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener"
debug="0"/>
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
debug="0"/>
<GlobalNamingResources>
<Environment name="simpleValue" type="java.lang.Integer" value="30"/>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved">
</Resource>
<ResourceParams name="UserDatabase">
<parameter>
<name>factory</name>
<value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
</parameter>
<parameter>
<name>pathname</name>
<value>conf/tomcat-users.xml</value>
</parameter>
</ResourceParams>
<ResourceParams name="UserDatabase">
<parameter>
<name>factory</name>
<value>org.apache.catalina.users.MemoryUserDatabaseFactory</value>
</parameter>
<parameter>
<name>pathname</name>
<value>conf/tomcat-users.xml</value>
</parameter>
</ResourceParams>
<Resource name="jdbc/NonTransactionalDataSource"
auth="Container"
type="javax.sql.DataSource"/>
<ResourceParams name="jdbc/NonTransactionalDataSource">
<parameter>
<name>factory</name>
<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>100</value>
</parameter>
<parameter>
<name>maxIdle</name>
<value>30</value>
</parameter>
<parameter>
<name>maxWait</name>
<value>10000</value>
</parameter>
<parameter>
<name>validationQuery</name>
<value>SELECT 1 FROM DUAL</value>
</parameter>
<parameter>
<name>username</name>
<value>appl_user</value>
</parameter>
<parameter>
<name>password</name>
<value>appl_user</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>oracle.jdbc.OracleDriver</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:oracle:thin:@[...]</value>
</parameter>
</ResourceParams>
<Resource name="jdbc/TransactionalDataSource"
auth="Container"
type="javax.sql.DataSource"/>
<ResourceParams name="jdbc/TransactionalDataSource">
<parameter>
<name>factory</name>
<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>100</value>
</parameter>
<parameter>
<name>maxIdle</name>
<value>30</value>
</parameter>
<parameter>
<name>maxWait</name>
<value>10000</value>
</parameter>
<parameter>
<name>validationQuery</name>
<value>SELECT 1 FROM DUAL</value>
</parameter>
<parameter>
<name>username</name>
<value>appl_user</value>
</parameter>
<parameter>
<name>password</name>
<value>appl_user</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>oracle.jdbc.OracleDriver</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:oracle:thin:@smedb501p:1521:wtop01</value>
</parameter>
</ResourceParams>
</GlobalNamingResources>
<Service name="Tomcat-Standalone">
<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
port="8029" minProcessors="50" maxProcessors="300"
enableLookups="true" redirectPort="8443"
acceptCount="10" debug="0" connectionTimeout="20000"
useURIValidationHack="false" disableUploadTimeout="true" />
<Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
port="8009" minProcessors="50" maxProcessors="300"
enableLookups="false" redirectPort="8443"
acceptCount="10" debug="0" connectionTimeout="0"
useURIValidationHack="false"
protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/>
<Engine name="Standalone" defaultHost="localhost" debug="0">
<Logger className="org.apache.catalina.logger.FileLogger"
prefix="catalina_log." suffix=".txt"
timestamp="true"/>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
debug="0" resourceName="UserDatabase"/>
<Host name="localhost" debug="0" appBase="webapps"
unpackWARs="true" autoDeploy="true" liveDeploy="false">
<Logger className="org.apache.catalina.logger.FileLogger"
directory="logs" prefix="localhost_log." suffix=".txt"
timestamp="true"/>
<Context path="/appl" reloadable="false" docBase="appl" >
<Logger className="org.apache.catalina.logger.SystemOutLogger"
verbosity="4" timestamp="true"/>
<ResourceLink global="jdbc/NonTransactionalDataSource"
name="jdbc/NonTransactionalDataSource" type="javax.sql.DataSource"/>
<ResourceLink global="jdbc/TransactionalDataSource"
name="jdbc/TransactionalDataSource" type="javax.sql.DataSource"/>
</Context>
</Host>
</Engine>
</Service>
</Server>
===================================================
Any help will be appreciated.
Thank you for your time.
Regards,
Vasily
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 4.1.13 dies every 3-4 days
Posted by Vasily Ivanov <ba...@gmail.com>.
This is an update from our system administrator:
"From what I can tell from the errors and the research I have done the
ajp_process_callback from the jk_ajp_common.c function is failing
because it is not recieving a close code from the apache instance -
possibly when a sender (browser) stopped sending data before the end -
certainly in a big post. This means that the client closed the browser
before all data has been set (sent). There is probably a setting
somewhere in tomcat that handles this.
This is the section of code from jk_ajp_common.c that produces this
error:
if ((len = ajp_read_fully_from_server(r, read_buf, len)) < 0) {
jk_log(l, JK_LOG_INFO,
"ERROR: receiving data from client failed. "
"Connection aborted or network problems\n");
return JK_CLIENT_ERROR;
}"
What is that "setting somewhere in tomcat"?
Thank you.
Regards,
Vasily
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org