endpoint connections constantly reconnecting

[David Nillesen <dn...@une.edu.au>]

Hi,

                I'm trying to track down some errors with our apache tomcat system and not having a great deal of luck.

                We had an issue where the mod_jk connectors would loose track of the tomcat connectors and just start refusing requests with error 500. This was with an apache front end and tomcat backend.

                We seem to have the majority of this sorted, so now I am trying to clean up the last few problems. Namely all these errors:

[Fri Aug 22 14:38:33.369 2008] [15192:2534538208] [info] ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:33.901 2008] [19082:2534538208] [info] ajp_send_request::jk_ajp_common.c (1347): (LOGIN-PROD) all endpoints are disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:33.941 2008] [15184:2534538208] [info] ajp_send_request::jk_ajp_common.c (1347): (LOGIN-PROD) all endpoints are disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:36.159 2008] [15243:2534538208] [info] ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are disconnected, detected by connect check (1), cping (0), send (0)
[Fri Aug 22 14:38:36.394 2008] [22157:2534538208] [info] ajp_send_request::jk_ajp_common.c (1347): (TLC-PROD) all endpoints are disconnected, detected by connect check (1), cping (0), send (0)

                These seem to be relatively benign, but they annoy me and as far as I tell should not occur.

                After a bit of reformatting of the mod_jk log I ended up with these snippet as an example:

[9641:2534538208] (MY-PROD) [Fri Aug 22 12:11:35.193 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:15:35.138 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:17:09.159 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:17:25.898 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:18:18.216 2008]
[9641:2534538208] (MY-PROD) [Fri Aug 22 12:18:48.942 2008]

                This seems to indicate that a single process 9461 is finding its endpoint to the MY-PROD worker disappearing on a regular basis. There are even drops ~20 secs apart.

As far as I can tell there should be 1 worker per apache process (I'm using prefork). We run 12 different workers for this single apache instance.  Does this mean 1 of each of the 12 workers can serve each Apache process at a time or is it 1 worker at a time total per apache process?

I could understand the number of disconnects of the apache processes have to keep swapping between workers, but it has me worried if there is no need to do this.

The configuration is setup such that apache should hold a pool open for 10 minutes at idle. Apache processes are not required to close they have a 0 max requests set. I have timeouts configured correctly I hope for the connectors, here are the relevant snippets:

# global configurations
worker.template.type=ajp13
worker.template.lbfactor=1
worker.template.connection_pool_timeout=600
worker.template.connect_timeout=5000
worker.template.prepost_timeout=5000

worker.MY-PROD.port=8010
worker.MY-PROD.host=127.0.0.1
worker.MY-PROD.reference=worker.template

                <Connector port="8010"
                        maxThreads="300"
                        minSpareThreads="80"
                        maxSpareThreads="50"
                        connectionTimeout="600000"
                        enableLookups="false"
                        redirectPort="-1"
                        protocol="AJP/1.3" />

I have also set JKShmSize to a much larger number so that there is no possibility of running out of space.  How many workers actually use this? I have 12 configured in my apache instance. Is this 12 total? Or is it 1 worker per process,  so a max of 254? Or is it the worst case of 12 worker definitions * 254 possible processes?

If there is also some sort of nuts and bolts how tomcat works at a lower level type document? Part of my problem may stem from the way I'm trying to guess underlying behaviour from config options.

Any help much appreciated,
Dave