You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ben Ricker <br...@wellinx.com> on 2002/12/23 22:33:00 UTC
Urgent - Issue with Tomcat/mod_jk?
I posted a message earlier which has not received any replies so I will
try another tact. The earlier mail is posted below this one.
I have a production issue that has arisen in a setup of two Apache
1.3.27/Redhat 7.3 servers load-balanced across 4 Tomcat 4.0.6/Sun
Solaris servers. Tomcat is running with Sun's 1.3 JVM.
We have gotten a lot of the following errors in the "engine" log for
Tomcat:
2002-12-23 15:21:56 Ajp13Processor[12009][16] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
We have also gotten calls from customers saying they are getting hanging
screens in our application. We see no anomolies on out database,
although earlier we did.
Can anyone tell me what the error above is saying? Is it not able to
write to the mod_jk? It looks like it is failing in some sort of closing
routine (org.apache.ajp.tomcat4.Ajp13Response.finishResponse). Could
this be a protocol issue?
I am getting heat from my bosses on this one and I need to say
SOMETHING! Please help if you can give me any clues or suggest
possibilities.
Thanks,
Ben Ricker
-----Earlier Post-----
We saw a strange production issue this morning that seems to be related
to Apache/mod_jk but I wanted to see if anyone can shed some lght on it.
First, the environment:
2 Redhat 7.3 Linux server talking to 2 Sun Solaris 8 Tomcat 4.0.6
servers. Each server has 2 separate Tomcat instances for a total of 4.
The Java uses Oracles JDBC drivers (not through Tomcat, but through a
custom connection pool) to talk to an Oracle DB runnin on AIX 4.x.
The first symptom was a huge spike in DB usage due to the creation of
around 35 pooled connections to the database, This led us to think
something happened on the database side, but it seems that the pool only
grew on one of the Tomcat instances; the other instances showed the
default pool size. The Oracle database did not show any locked tables or
huge, hanging queries.
The only problems I see in logs are related to mod_jk and Tomcat.
First in the Tomcat engine log,I see:
2002-12-23 09:07:14 Ajp13Connector[12009] No processor available, rejecting this connection
There are hundreds of these messages. Then I start seeing this message
in the engine log:
2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
About a minute after, in the mod_jk logs, I see:
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (679)]: ajp_connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (1041)]: Error reading reply
[Mon Dec 23 09:08:29 2002] [jk_ajp_common.c (1178)]: In jk_endpoint_t::service, ajp_get_reply failed in send loop 0
This goes on for about 20 seconds before I see a huge number of the
following:
[Mon Dec 23 09:08:49 2002] [jk_ajp_common.c (961)]: Error ajp_process_callback - write failed
I am at a loss at what might cause this. Could it be related to
soemthing in Tomcat? What exactly does this sequence of events tell me?
Any light one can shed would be greatly appreciated.
Ben Ricker
--
Ben Ricker <br...@wellinx.com>
Wellinx.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Urgent - Issue with Tomcat/mod_jk?
Posted by Ben Ricker <br...@wellinx.com>.
I had suspected that in the problem earlier in the day (the part of the
email I posted at the end if the last email you replied to). However,
what did not make sense was the connection pool woth the database also
going up.
If I understand the stuff below, after 85 concurrent connections (max
connections + accept count) I would start getting the "Out of
Processors" error. However, I cannot see how this would cause the
DATABASE connection pool to grow as large as it did (we usually handle
50k connections a business day with 5 pooled connections; during the
earlier problem, the database connections went to *30*, our maximum).
Additionally, why would we contine to get the following error:
2002-12-23 09:07:28 Ajp13Processor[12009][18] process: invoke
java.io.IOException: Broken pipe
at java.net.SocketOutputStream.socketWrite(Native Method)
at java.net.SocketOutputStream.write(SocketOutputStream.java:91)
at org.apache.ajp.Ajp13.send(Ajp13.java:525)
at org.apache.ajp.RequestHandler.finish(RequestHandler.java:501)
at org.apache.ajp.Ajp13.finish(Ajp13.java:395)
at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:196)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:464)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:551)
at java.lang.Thread.run(Thread.java:479)
without seeing the "Out of Processor" msgs?
I may be looking at two independent problems: one a scalability issue
with the ajp13 processors (which are at the ddefault setting, btw) and
some other issue I am in the dark about.
Anyway, these questions are rhetorical. Thanks for the reply and do not
feel obliged to answer back. I need to start putting pressure on the
developers to help me out here.
Thanks again,
Ben Ricker
On Mon, 2002-12-23 at 16:07, Mark Eggers wrote:
> Ben,
>
> Disclaimer:
>
> I'm not a Tomcat developer, but I do use it to develop
> software and integrate applications.
>
> In $CATALINA_HOME/conf/server.xml you should see an
> entry similar to the following:
>
> <!-- Define an AJP 1.3 Connector on port 8009 -->
> <Connector
> className="org.apache.ajp.tomcat4.Ajp13Connector"
> port="8009"
> minProcessors="5"
> maxProcessors="75"
> acceptCount="10" debug="0"/>
>
> I'm using 4.1.18, so your entry may be a bit
> different. It looks like from your error messages
> that you may be running out of processors.
>
> The following information is taken from Tomcat's
> 4.1.18 documentation. If you have the documentation
> installed somewhere, the URL is:
>
> http://localhost/tomcat-docs/config/jk.html
>
> acceptCount:
>
> The maximum queue length for incoming connection
> requests when all possible request processing threads
> are in use. Any requests received when the queue is
> full will be refused. The default value is 10.
>
> maxProcessors:
>
> The maximum number of request processing threads to be
> created by this Connector, which therefore determines
> the maximum number of simultaneous requests that can
> be handled. If not specified, this attribute is set to
> 20.
>
> NOTE:For Apache 1.3 on Unix there is a 1 to 1 mapping
> between httpd processes and Ajp13Processors. You must
> configure maxProcessors to be greater than or equal to
> the maximum number of httpd processes your Apache web
> server spawns.
>
> minProcessors:
>
> The number of request processing threads that will be
> created when this Connector is first started. This
> attribute should be set to a value smaller than that
> set for maxProcessors. The default value is 5.
>
> I hope this gets you started on a productive path.
>
> /mde/
>
> just my two cents . . . .
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
--
Ben Ricker <br...@wellinx.com>
Wellinx.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: Urgent - Issue with Tomcat/mod_jk?
Posted by Mark Eggers <it...@yahoo.com>.
Ben,
Disclaimer:
I'm not a Tomcat developer, but I do use it to develop
software and integrate applications.
In $CATALINA_HOME/conf/server.xml you should see an
entry similar to the following:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector
className="org.apache.ajp.tomcat4.Ajp13Connector"
port="8009"
minProcessors="5"
maxProcessors="75"
acceptCount="10" debug="0"/>
I'm using 4.1.18, so your entry may be a bit
different. It looks like from your error messages
that you may be running out of processors.
The following information is taken from Tomcat's
4.1.18 documentation. If you have the documentation
installed somewhere, the URL is:
http://localhost/tomcat-docs/config/jk.html
acceptCount:
The maximum queue length for incoming connection
requests when all possible request processing threads
are in use. Any requests received when the queue is
full will be refused. The default value is 10.
maxProcessors:
The maximum number of request processing threads to be
created by this Connector, which therefore determines
the maximum number of simultaneous requests that can
be handled. If not specified, this attribute is set to
20.
NOTE:For Apache 1.3 on Unix there is a 1 to 1 mapping
between httpd processes and Ajp13Processors. You must
configure maxProcessors to be greater than or equal to
the maximum number of httpd processes your Apache web
server spawns.
minProcessors:
The number of request processing threads that will be
created when this Connector is first started. This
attribute should be set to a value smaller than that
set for maxProcessors. The default value is 5.
I hope this gets you started on a productive path.
/mde/
just my two cents . . . .
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>