You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by János Löbb <ja...@yale.edu> on 2009/03/20 18:48:53 UTC
reversed proxy stopped working with tomcat cluster
Hi,
I have two real machines. One of them is a Windows XP running Apache
2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The
other one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26
and Tomcat 6.0.16.
The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, -
node1,2,3. Altogether 4 tomcats and they form a cluster. Here is one
server .xml from node3.
<snip>
<?xml version='1.0' encoding='utf-8'?>
<Server port="8205" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener
className
="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8280" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8643" />
<Connector port="8209" protocol="AJP/1.3" redirectPort="8643" />
<Engine name="Catalina" defaultHost="localhost" jvmRoute="node3">
<Cluster
className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
</Host>
</Engine>
</Service>
</Server>
<snip>
In front of these two machines is an Apache 2.2.10 in a VMWare Fusion
virtual Machine on Windows XP in the Mac and configured as a Reverse
proxy
Here is the reverse proxy config from the httpd.conf
<snip>
ProxyRequests Off
<Proxy balancer://pathCluster>
BalancerMember http://bml0039.yalepath.org loadfactor=10
BalancerMember http://bml0073.yalepath.org loadfactor=10
ProxySet lbmethod=bytraffic
</Proxy>
ProxyPass /tc/ balancer://pathCluster/
ProxyPassReverse /tc/ balancer://pathCluster/
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Allow from .yalepath.org
</Location>
</snip>
Here is the workers.property file:
<snip>
bml0073:local janos$ cat apache2/conf/workers.properties
worker.list = lb,jkstatus
worker.lb.type=lb
worker.lb.balance_workers=node1,node2,node3,node4
worker.lb.sticky_session = True
worker.lb.sticky_session_force = False
worker.jkstatus.type=status
worker.node1.type = ajp13
worker.node1.host = localhost
worker.node1.port = 8009
worker.node1.lbfactor = 1
worker.node1.redirect=node2
worker.node2.type = ajp13
worker.node2.host = localhost
worker.node2.port = 8109
worker.node2.lbfactor = 1
worker.node2.redirect=node3
worker.node3.type = ajp13
worker.node3.host = localhost
worker.node3.port = 8209
worker.node3.lbfactor = 1
worker.node3.redirect=node4
worker.node4.type = ajp13
worker.node4.host = bml0039.yalepath.org
worker.node4.port = 8309
worker.node4.lbfactor = 1
worker.node4.redirect=node1
</snip>
Similar workers.properies on the XP machine but the hosts are
logically different
Well, this setup worked in December. The only change I made is that
the sticky parameters now assigned to the loadbalancing worker 'lb'
and not to the individual real workers. /Thanks to pointing that
out :) / There were some updates to the Mac OS and also to the VmWare
Fussion. I fired up this cluster yesterday and I realized it does not
work anymore.
Here is the access_log from the reverse proxy:
<snip>
10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] "GET /tc/CassetteLabeler
HTTP/1.1" 302 -
10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /tc/CassetteLabeler/
HTTP/1.1" 200 1506
10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler/
pathology.labeler.labeler.Labeler/Labeler.html HTTP/1.1" 404 260
</snip>
here are the entries from the error log:
<snip>
[Fri Mar 20 12:08:30 2009] [notice] Apache/2.2.10 (Win32) configured
-- resuming normal operations
[Fri Mar 20 12:08:30 2009] [notice] Server built: Oct 10 2008 12:39:04
[Fri Mar 20 12:08:31 2009] [notice] Parent: Created child process 2564
[Fri Mar 20 12:08:31 2009] [notice] Child 2564: Child process is running
[Fri Mar 20 12:08:31 2009] [notice] Child 2564: Acquired the start
mutex.
[Fri Mar 20 12:08:31 2009] [notice] Child 2564: Starting 64 worker
threads.
[Fri Mar 20 12:08:31 2009] [notice] Child 2564: Starting thread to
listen on port 80.
[Fri Mar 20 12:09:32 2009] [error] [client 10.84.2.195] File does not
exist: C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/
favicon.ico
[Fri Mar 20 12:09:35 2009] [error] [client 10.84.2.195] File does not
exist: C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/
favicon.ico
[Fri Mar 20 12:09:49 2009] [error] [client 10.84.2.195] File does not
exist: C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/
CassetteLabeler, referer: http://bml0195.yalepath.org/tc/CassetteLabeler/
</snip>
The request from the reverse proxy did go to the XP machine:
Here are the log entries from the access log:
<snip>
10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler
HTTP/1.1" 302 -
</snip>
Here are the log entries from the error_log:
<snip>
[Fri Mar 20 12:28:36 2009] [error] [client 10.84.2.73] File does not
exist: C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/
favicon.ico, referer: http://bml0039.yalepath.org/CassetteLabeler/
</snip>
Here are the entries from mod_jk.log of this XP machine:
<snip>
[Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c
(682): Attempting to map URI '/CassetteLabeler' from 5 maps
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(503): Attempting to map context URI '/examples/servlets/*=lb' source
'JkMount'
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(503): Attempting to map context URI '/examples/jsp/*=lb' source
'JkMount'
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(503): Attempting to map context URI '/jkmanager/*=jkstatus' source
'JkMount'
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(503): Attempting to map context URI '/examples/*=lb' source 'JkMount'
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(503): Attempting to map context URI '/CassetteLabeler*=lb' source
'JkMount'
[Fri Mar 20 12:09:49.270 2009] [852:1808] [debug] jk_uri_worker_map.c
(516): Found a wildchar match '/CassetteLabeler*=lb'
[Fri Mar 20 12:09:49.410 2009] [852:1808] [debug] mod_jk.c (2222):
Into handler jakarta-servlet worker=lb r->proxyreq=0
[Fri Mar 20 12:09:49.426 2009] [852:1808] [debug] jk_worker.c (115):
found a worker lb
[Fri Mar 20 12:09:49.426 2009] [852:1808] [debug] jk_worker.c (323):
Maintaining worker lb
[Fri Mar 20 12:09:49.426 2009] [852:1808] [debug] jk_lb_worker.c
(556): decay with 2^1168
[Fri Mar 20 12:09:49.426 2009] [852:1808] [debug] jk_worker.c (292):
Found worker type 'lb'
[Fri Mar 20 12:09:49.457 2009] [852:1808] [debug] mod_jk.c (888):
Service protocol=HTTP/1.1 method=GET host=(null) addr=10.84.2.195
name=bml0039.yalepath.org port=80 auth=(null) user=(null)
laddr=10.84.2.39 raddr=10.84.2.195 uri=/CassetteLabeler
[Fri Mar 20 12:09:49.473 2009] [852:1808] [debug] jk_lb_worker.c
(940): service sticky_session=1 id='empty'
[Fri Mar 20 12:09:49.473 2009] [852:1808] [debug] jk_lb_worker.c
(830): found best worker node3 (node3) using method 'Request'
[Fri Mar 20 12:09:49.473 2009] [852:1808] [debug] jk_lb_worker.c
(962): service worker=node3 route=node3
[Fri Mar 20 12:09:49.488 2009] [852:1808] [debug] jk_ajp_common.c
(2579): acquired connection pool slot=0
[Fri Mar 20 12:09:49.488 2009] [852:1808] [debug] jk_ajp_common.c
(553): ajp marshaling done
[Fri Mar 20 12:09:49.488 2009] [852:1808] [debug] jk_ajp_common.c
(2050): processing node3 with 2 retries
[Fri Mar 20 12:09:49.488 2009] [852:1808] [debug] jk_ajp_common.c
(1352): (node3) all endpoints are disconnected, detected by connect
check (0), cping (0), send (0)
[Fri Mar 20 12:09:49.504 2009] [852:1808] [debug] jk_connect.c (448):
socket TCP_NODELAY set to On
[Fri Mar 20 12:09:49.504 2009] [852:1808] [debug] jk_connect.c (548):
trying to connect socket 392 to 10.84.2.73:8209
[Fri Mar 20 12:09:49.504 2009] [852:1808] [debug] jk_connect.c (574):
socket 392 connected to 10.84.2.73:8209
[Fri Mar 20 12:09:49.504 2009] [852:1808] [debug] jk_ajp_common.c
(878): Connected socket 392 to (10.84.2.73:8209)
[Fri Mar 20 12:09:49.504 2009] [852:1808] [debug] jk_ajp_common.c
(934): sending to ajp13 pos=4 len=545 max=8192
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0000 12 34 02 1D 02 02 00 08 48 54 54 50 2F 31 2E 31 - .
4......HTTP/1.1
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0010 00 00 10 2F 43 61 73 73 65 74 74 65 4C 61 62 65 - .../
CassetteLabe
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0020 6C 65 72 00 00 0B 31 30 2E 38 34 2E 32 2E 31 39 -
ler...10.84.2.19
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0030 35 00 FF FF 00 14 62 6D 6C 30 30 33 39 2E 79 61 -
5.....bml0039.ya
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0040 6C 65 70 61 74 68 2E 6F 72 67 00 00 50 00 00 0B -
lepath.org..P...
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0050 A0 0B 00 14 62 6D 6C 30 30 33 39 2E 79 61 6C 65
- ....bml0039.yale
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0060 70 61 74 68 2E 6F 72 67 00 A0 0E 00 5A 4D 6F 7A -
path.org....ZMoz
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0070 69 6C 6C 61 2F 35 2E 30 20 28 57 69 6E 64 6F 77 - illa/
5.0.(Window
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0080 73 3B 20 55 3B 20 57 69 6E 64 6F 77 73 20 4E 54 -
s;.U;.Windows.NT
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0090 20 35 2E 31 3B 20 65 6E 2D 55 53 3B 20 72 76 3A - .
5.1;.en-US;.rv:
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00a0 31 2E 39 2E 30 2E 35 29 20 47 65 63 6B 6F 2F 32 -
1.9.0.5).Gecko/2
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00b0 30 30 38 31 32 30 31 32 32 20 46 69 72 65 66 6F -
008120122.Firefo
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00c0 78 2F 33 2E 30 2E 35 00 A0 01 00 3F 74 65 78 74 - x/
3.0.5....?text
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00d0 2F 68 74 6D 6C 2C 61 70 70 6C 69 63 61 74 69 6F - /
html,applicatio
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00e0 6E 2F 78 68 74 6D 6C 2B 78 6D 6C 2C 61 70 70 6C - n/
xhtml+xml,appl
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 00f0 69 63 61 74 69 6F 6E 2F 78 6D 6C 3B 71 3D 30 2E -
ication/xml;q=0.
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0100 39 2C 2A 2F 2A 3B 71 3D 30 2E 38 00 00 0F 41 63 - 9,*/
*;q=0.8...Ac
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0110 63 65 70 74 2D 4C 61 6E 67 75 61 67 65 00 00 0E - cept-
Language...
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0120 65 6E 2D 75 73 2C 65 6E 3B 71 3D 30 2E 35 00 00 - en-
us,en;q=0.5..
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0130 0F 41 63 63 65 70 74 2D 45 6E 63 6F 64 69 6E 67
- .Accept-Encoding
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0140 00 00 0C 67 7A 69 70 2C 64 65 66 6C 61 74 65 00
- ...gzip,deflate.
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0150 00 0E 41 63 63 65 70 74 2D 43 68 61 72 73 65 74
- ..Accept-Charset
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0160 00 00 1E 49 53 4F 2D 38 38 35 39 2D 31 2C 75 74
- ...ISO-8859-1,ut
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0170 66 2D 38 3B 71 3D 30 2E 37 2C 2A 3B 71 3D 30 2E -
f-8;q=0.7,*;q=0.
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0180 37 00 00 0F 58 2D 46 6F 72 77 61 72 64 65 64 2D -
7...X-Forwarded-
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0190 46 6F 72 00 00 0B 31 30 2E 38 34 2E 32 2E 31 39 -
For...10.84.2.19
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01a0 35 00 00 10 58 2D 46 6F 72 77 61 72 64 65 64 2D -
5...X-Forwarded-
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01b0 48 6F 73 74 00 00 14 62 6D 6C 30 31 39 35 2E 79 -
Host...bml0195.y
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01c0 61 6C 65 70 61 74 68 2E 6F 72 67 00 00 12 58 2D -
alepath.org...X-
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01d0 46 6F 72 77 61 72 64 65 64 2D 53 65 72 76 65 72 -
Forwarded-Server
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01e0 00 00 1E 50 61 74 68 2D 4A 61 6E 6F 73 2D 46 75
- ...Path-Janos-Fu
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 01f0 73 69 6F 6E 2E 79 61 6C 65 70 61 74 68 2E 6F 72 -
sion.yalepath.or
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0200 67 00 A0 06 00 0A 4B 65 65 70 2D 41 6C 69 76 65 -
g.....Keep-Alive
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0210 00 A0 08 00 01 30 00 06 00 05 6E 6F 64 65 33 00 - .....
0....node3.
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(934): 0220 FF 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
- ................
[Fri Mar 20 12:09:49.520 2009] [852:1808] [debug] jk_ajp_common.c
(1395): (node3) request body to send 0 - request body to resend 0
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): received from ajp13 pos=0 len=83 max=8192
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0000 04 01 2E 00 11 4D 6F 76 65 64 20 54 65 6D 70 6F
- .....Moved.Tempo
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0010 72 61 72 69 6C 79 00 00 01 00 08 4C 6F 63 61 74 -
rarily.....Locat
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0020 69 6F 6E 00 00 2C 68 74 74 70 3A 2F 2F 62 6D 6C -
ion..,http://bml
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0030 30 30 33 39 2E 79 61 6C 65 70 61 74 68 2E 6F 72 -
0039.yalepath.or
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0040 67 2F 43 61 73 73 65 74 74 65 4C 61 62 65 6C 65 - g/
CassetteLabele
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0050 72 2F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -
r/..............
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(608): status = 302
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(615): Number of headers is = 1
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(671): Header[0] [Location] = [http://bml0039.yalepath.org/CassetteLabeler/
]
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): received from ajp13 pos=0 len=2 max=8192
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1117): 0000 05 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00
- ................
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(1661): AJP13 protocol: Reuse is OK
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(691): (node3) resetting endpoint with sd = 392
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
(2522): recycling connection pool slot=0 for worker node3
[Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] mod_jk.c (2348):
Service finished with status=302 for worker=lb
</snip>
Both balance members - XP or Mac - are working perfectly when I
request the application from them via apache like:
http://bml0073.yalepath.org/CassetteLabeler
or
http://bml0039.yalepath.org/CasetteLabeler
Here is the catalina daily log from node3
<snip>
bml0073:local janos$ cat tomcat/node3/logs/catalina.2009-03-20.log
Mar 20, 2009 11:13:47 AM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8280
Mar 20, 2009 11:13:48 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Mar 20, 2009 11:13:48 AM
org.apache.catalina.ha.session.JvmRouteBinderValve stop
INFO: JvmRouteBinderValve stopped
Mar 20, 2009 11:13:48 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/Dictation] expiring sessions upon shutdown
Mar 20, 2009 11:13:48 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/examples] expiring sessions upon shutdown
Mar 20, 2009 11:13:48 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/CassetteLabeler] expiring sessions upon
shutdown
Mar 20, 2009 11:13:49 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/host-manager] expiring sessions upon shutdown
Mar 20, 2009 11:13:49 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/docs] expiring sessions upon shutdown
Mar 20, 2009 11:13:49 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#] expiring sessions upon shutdown
Mar 20, 2009 11:13:49 AM org.apache.catalina.ha.session.DeltaManager
stop
INFO: Manager [localhost#/manager] expiring sessions upon shutdown
Mar 20, 2009 11:13:49 AM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8280
Mar 20, 2009 11:30:40 AM org.apache.catalina.core.AprLifecycleListener
init
INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the
java.library.path: .:/Library/Java/Extensions:/System/Library/Java/
Extensions:/usr/lib/java
Mar 20, 2009 11:30:40 AM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8280
Mar 20, 2009 11:30:40 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 769 ms
Mar 20, 2009 11:30:40 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Mar 20, 2009 11:30:40 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.18
Mar 20, 2009 11:30:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
start
INFO: Cluster is about to start
Mar 20, 2009 11:30:40 AM
org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/10.84.2.73:4002
Mar 20, 2009 11:30:40 AM
org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Mar 20, 2009 11:30:40 AM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:4
Mar 20, 2009 11:30:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 73}:4001,{10, 84, 2, 73},4001, alive=10515,id={-78 -117 35 -119 -78
-99 73 -112 -107 43 127 -30 -59 -62 73 -72 }, payload={}, command={},
domain={}, ]
Mar 20, 2009 11:30:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 73}:4000,{10, 84, 2, 73},4000, alive=18517,id={-47 109 37 -40 96 28
76 -120 -112 66 -110 11 -96 16 -50 -74 }, payload={}, command={},
domain={}, ]
Mar 20, 2009 11:30:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster
memberAdded
INFO: Replication member
added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 39}:4000,{10, 84, 2, 39},4000, alive=68758042,id={13 -96 -89 -87 2
-80 68 106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]
Mar 20, 2009 11:30:41 AM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:4
Mar 20, 2009 11:30:41 AM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership,
start level:8
Mar 20, 2009 11:30:41 AM org.apache.catalina.tribes.io.BufferPool
getBufferPool
INFO: Created a buffer pool with max size:104857600 bytes of
type:org.apache.catalina.tribes.io.BufferPool15Impl
Mar 20, 2009 11:30:42 AM
org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:8
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /CassetteLabeler to cluster element Engine with
name Catalina
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /CassetteLabeler
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/CassetteLabeler], requesting session
state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10,
84, 2, 39}:4000,{10, 84, 2, 39},4000, alive=68760042,id={13 -96 -89
-87 2 -80 68 106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/CassetteLabeler]; session state send at
3/20/09 11:30 AM received in 118 ms.
Mar 20, 2009 11:30:43 AM org.apache.catalina.startup.HostConfig
deployWAR
INFO: Deploying web application archive Dictation.war
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /Dictation to cluster element Engine with name
Catalina
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /Dictation
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/Dictation], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2, 39}:
4000,{10, 84, 2, 39},4000, alive=68760542,id={13 -96 -89 -87 2 -80 68
106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/Dictation]; session state send at 3/20/09
11:30 AM received in 104 ms.
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /docs to cluster element Engine with name
Catalina
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /docs
Mar 20, 2009 11:30:43 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/docs], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2, 39}:
4000,{10, 84, 2, 39},4000, alive=68761042,id={13 -96 -89 -87 2 -80 68
106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/docs]; session state send at 3/20/09 11:30
AM received in 102 ms.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /examples to cluster element Engine with name
Catalina
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /examples
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/examples], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2, 39}:
4000,{10, 84, 2, 39},4000, alive=68761042,id={13 -96 -89 -87 2 -80 68
106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/examples]; session state send at 3/20/09
11:30 AM received in 101 ms.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /host-manager to cluster element Engine with
name Catalina
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /host-manager
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/host-manager], requesting session state
from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84,
2, 39}:4000,{10, 84, 2, 39},4000, alive=68761542,id={13 -96 -89 -87 2
-80 68 106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/host-manager]; session state send at 3/20/09
11:30 AM received in 101 ms.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager /manager to cluster element Engine with name
Catalina
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /manager
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#/manager], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2, 39}:
4000,{10, 84, 2, 39},4000, alive=68761542,id={13 -96 -89 -87 2 -80 68
106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/manager]; session state send at 3/20/09
11:30 AM received in 102 ms.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Register manager to cluster element Engine with name Catalina
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
getAllClusterSessions
WARNING: Manager [localhost#], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 84, 2, 39}:
4000,{10, 84, 2, 39},4000, alive=68761542,id={13 -96 -89 -87 2 -80 68
106 -69 -29 -48 94 10 57 8 -20 }, payload={}, command={},
domain={}, ]. This operation will timeout if no session state has been
received within 60 seconds.
Mar 20, 2009 11:30:44 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#]; session state send at 3/20/09 11:30 AM
received in 101 ms.
Mar 20, 2009 11:30:44 AM
org.apache.catalina.ha.session.JvmRouteBinderValve start
INFO: JvmRouteBinderValve started
Mar 20, 2009 11:30:44 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8280
Mar 20, 2009 11:30:44 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8209
Mar 20, 2009 11:30:44 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/20 config=null
Mar 20, 2009 11:30:44 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4550 ms
</snip>
I have a production environment - just with 3 Macs and just one tomcat
per machine forming a cluster - where a very similar configuration
runs fine. The only difference is that the sticky parameters are NOT
with the load balancer but with the real workers in the
workers.properties file, and the reverse proxy configuration somewhat
different. Here it is:
<snip>
<IfModule mod_proxy_balancer.c>
ProxyPass /apps/ balancer://balancer-group/ stickysession=JSESSIONID
ProxyPassReverse /apps/ balancer://balancer-group/
<Proxy "balancer://balancer-group">
BalancerMember http://pathsrv1.yalepath.org loadfactor=10
BalancerMember http://raid2b.yalepath.org loadfactor=10
BalancerMember http://reptile.yalepath.org loadfactor=10
</Proxy>
</IfModule>
</snip>
That is there is this stickysession=JSESSIONID in the ProxyPass
statement and it was generated by the Server Admin tool as the reverse
proxy was configured via the GUI.
Thanks ahead,
János
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
János,
On 3/25/2009 11:53 AM, János Löbb wrote:
> In our production environment we have 3 Xserves, all running apache2 ,
> mod+jk and Tomcat. There are about 5-6 web applications running on
> these 3 servers standalone, -non-clustered - and there will be more.
> The there Xserves are doing other things too, not just web serving.
> That is the reason I cannot reduce the number of apache2 instances.
> They grandfathered in themselves :)
Just because they're there doesn't mean you have to use them. As long as
the Tomcat ports are directly accessible from the load balancer, you can
simply skip the intermediate httpd instances.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknKbBgACgkQ9CaO5/Lv0PBk7gCdG8EQqggQ1ngeRcHMUjTH+M55
c/MAoLTqMlyXNOCmj7Uu27Z1AYjd+hp4
=9lj/
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Posted by János Löbb <ja...@yale.edu>.
Hi Chris,
In our production environment we have 3 Xserves, all running apache2 ,
mod+jk and Tomcat. There are about 5-6 web applications running on
these 3 servers standalone, -non-clustered - and there will be more.
The there Xserves are doing other things too, not just web serving.
That is the reason I cannot reduce the number of apache2 instances.
They grandfathered in themselves :)
This CassetteLabeler app is our first application that needs to be up
all the time and that is why we turned to clustering, using the
existing setups on these servers and hoping that just by small config
changes here and there we can get our clustered app runing without
worrying about the non-clustered apps. You know, the goat has to have
a full stomach and the cabbage had to remain too :) . Not to destroy
accidently anything in the production environment, I created a small
test environment, simulating our production environment. Because I do
not have 3 machines, I used the VmWare Fussion on my Mac with Windows
XP to create the third machine, and to simplify things I do not run
mod_jk and Tomcat on it just Apache2.
The four Tomcats are forming a Tomcat Cluster. Because the individual
Apaches have to communicate to their own Tomcats via their own mod_jk
for the non-clustered webapps /non-clustered because they cannot be
serialized/ to reach those apps, the users connect via:
http://balancemember1.yalepath.org/Standalone1
kind of URL and it goes through port 80 to mod_jk and mod_jk passes it
via ajp to to the appropriate tomcat. We need to preserve this
behavior for the clustered app, that is why in the reversed proxy
configuration the default port 80 is used. I do not think that I can
changed that one.
The reason for the reverse proxy is, that two of the machines in
production are behind a Pix firewall and the users of the clustered
app can get to it by hitting the reverse proxy.
So in theory a user would hit a url like:
http://reverse_proxy.yalepath.org/tc/CassetteLabeler
The reverse proxy using mod_proxy, mod_proxy_balancer, mod_proxy_http
would transfer it to one of the balancemembers via port 80 by the
selected lbmethod. It would arrive there as:
http://balancememberX.yalepath.org/CassetteLabeler
The selected balance member seeing that there is a JkMount for that
request would involve mod_jk and mod_jk would forward it via the ajp
connector to one of the selected tomcats from the cluster. /Or that
is how I imagine it :)/
Looks like the initial request is received by the reverse proxy. It
selects a balance member and sends the request to it correctly, that
is stripping off the /tc. Tha balancemember involves mod_jk and
mod_jk forwards it to one of the clustered tomcats. Looking the debug
info tomcat is doing its job and even asks for the body and receives
it. Then tomcat is closing the connection to the balance member.
That is where things get out of control. Then the selected balance
member instead of continue with the earlier selected tomcat instance -
although stickiness is specified - selects another tomcat node and
sends the SAME info to it. This second tomcat does exactly as the
first one did and it closes the connection. At that point the
balancemember /or the reverse proxy/ sends the non-formated URL from
the reverse proxy to the Apache of the second tomcat node and of
course that cannot be resolved so I get a 404 error.
So, I am looking why stickiness is not working, why the selected
balancemember is not satisfied with the firstly selected tomcat node
and just finishing the whole session with it, so the right info can
flow back on the chain to the clients browser and paint the first form
of the web app.
Now I configured stickyness even in ProxyPass on the reverse proxy,
but no cigar.
ProxyRequests Off
<Proxy balancer://pathCluster>
BalancerMember http://bml0039.yalepath.org loadfactor=10
BalancerMember http://bml0073.yalepath.org loadfactor=10
ProxySet lbmethod=bytraffic
</Proxy>
ProxyPass /tc/ balancer://pathCluster/ stickysession=JSESSIONID
ProxyPassReverse /tc/ balancer://pathCluster/
#ProxyHTMLLogVerbose On
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Allow from .yalepath.org
</Location>
I would like to emphasize that when I request the web app from the
individual balance members using of course port 80, like:
http://balancemember1.yalepath.org/CassetteLabeler
or
http://balancemember2.yalepath.org/Cassettelabeler
, everything is working right. The problem is somewhere as the info
is flowing back in the chain to the reverse proxy.
Thanks ahead,
János
On Mar 25, 2009, at 9:33 AM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> János,
>
> On 3/20/2009 1:48 PM, János Löbb wrote:
>> <Connector port="8280" protocol="HTTP/1.1"
>> connectionTimeout="20000"
>> redirectPort="8643" />
>> <Connector port="8209" protocol="AJP/1.3" redirectPort="8643" />
>
> [snip]
>
>> ProxyRequests Off
>> <Proxy balancer://pathCluster>
>> BalancerMember http://bml0039.yalepath.org loadfactor=10
>> BalancerMember http://bml0073.yalepath.org loadfactor=10
>
> Note that you have configured your Tomcat to listen for HTTP on port
> 8280, but your <Proxy> configuration points to port 80. This isn't
> going
> to work.
>
>> Here is the workers.property file:
>>
>> <snip>
>> bml0073:local janos$ cat apache2/conf/workers.properties
>> worker.list = lb,jkstatus
>
>
> This looks like workers.properties, which is a mod_jk thing. You are
> using mod_proxy_http. This isn't going to work for two reasons:
>
> 1. mod_proxy_http uses HTTP as the protocol and mod_jk uses AJP
> 2. mod_proxy_http doesn't read workers.properties, so the file
> is not relevant
>
>> Similar workers.properies on the XP machine but the hosts are
>> logically
>> different
>
> If your Windows XP machine runs Tomcat only, then the presence of a
> workers.properties file is not relevant.
>
>> Well, this setup worked in December.
>
> Very unlikely.
>
>> The request from the reverse proxy did go to the XP machine:
>> Here are the log entries from the access log:
>>
>> <snip>
>> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler
>> HTTP/1.1" 302 -
>> </snip>
>
> Wait... are you running 3 copies of Apache httpd? No wonder there's
> tons
> and tons configuration. You only need one single httpd instance: the
> one
> doing the load-balancing. Configure /all/ of the TC instances that are
> part of the balancing setup in that one httpd. What you've done is set
> up load balancing across two httpd instances that then load-balance
> to a
> set of webapp servers. I can only think of a single reason to ever do
> that (you want lopsided load balancing) but it still doesn't make
> any sense.
>
> Lose the additional httpd instances and simplify your life, man. Then
> there won't be so much confusion over port numbers, forwarding,
> proxying, etc.
>
>> Here are the entries from mod_jk.log of this XP machine:
>>
>> <snip>
>> [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c
>> (682): Attempting to map URI '/CassetteLabeler' from 5 maps
>
> Ugh. Okay, so you are using mod_proxy_http for the main httpd and
> mod_jk
> from the other ones. That's not confusing at all.
>
> I would highly recommend that you simplify your configuration unless
> you
> have a real need to complicate it.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAknKMrUACgkQ9CaO5/Lv0PBjrACgvAP1zKwFa2wYlx7O4cvPx7DC
> JhAAoLd4B7HnrjSX+8f7IcoivwJcpiDs
> =/8Rs
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
János,
On 3/20/2009 1:48 PM, János Löbb wrote:
> <Connector port="8280" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8643" />
> <Connector port="8209" protocol="AJP/1.3" redirectPort="8643" />
[snip]
> ProxyRequests Off
> <Proxy balancer://pathCluster>
> BalancerMember http://bml0039.yalepath.org loadfactor=10
> BalancerMember http://bml0073.yalepath.org loadfactor=10
Note that you have configured your Tomcat to listen for HTTP on port
8280, but your <Proxy> configuration points to port 80. This isn't going
to work.
> Here is the workers.property file:
>
> <snip>
> bml0073:local janos$ cat apache2/conf/workers.properties
> worker.list = lb,jkstatus
This looks like workers.properties, which is a mod_jk thing. You are
using mod_proxy_http. This isn't going to work for two reasons:
1. mod_proxy_http uses HTTP as the protocol and mod_jk uses AJP
2. mod_proxy_http doesn't read workers.properties, so the file
is not relevant
> Similar workers.properies on the XP machine but the hosts are logically
> different
If your Windows XP machine runs Tomcat only, then the presence of a
workers.properties file is not relevant.
> Well, this setup worked in December.
Very unlikely.
> The request from the reverse proxy did go to the XP machine:
> Here are the log entries from the access log:
>
> <snip>
> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler
> HTTP/1.1" 302 -
> </snip>
Wait... are you running 3 copies of Apache httpd? No wonder there's tons
and tons configuration. You only need one single httpd instance: the one
doing the load-balancing. Configure /all/ of the TC instances that are
part of the balancing setup in that one httpd. What you've done is set
up load balancing across two httpd instances that then load-balance to a
set of webapp servers. I can only think of a single reason to ever do
that (you want lopsided load balancing) but it still doesn't make any sense.
Lose the additional httpd instances and simplify your life, man. Then
there won't be so much confusion over port numbers, forwarding,
proxying, etc.
> Here are the entries from mod_jk.log of this XP machine:
>
> <snip>
> [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c
> (682): Attempting to map URI '/CassetteLabeler' from 5 maps
Ugh. Okay, so you are using mod_proxy_http for the main httpd and mod_jk
from the other ones. That's not confusing at all.
I would highly recommend that you simplify your configuration unless you
have a real need to complicate it.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAknKMrUACgkQ9CaO5/Lv0PBjrACgvAP1zKwFa2wYlx7O4cvPx7DC
JhAAoLd4B7HnrjSX+8f7IcoivwJcpiDs
=/8Rs
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Posted by János Löbb <ja...@yale.edu>.
Hi Rainer,
Well, I did make some "bloody operation" by eliminating the VmWare
machine as the reverse proxy on my Mac. /Following Chris advise. :)/
Because apache2 on my Mac was not compiled with --enable-proxy, etc, I
downloaded the version 2.2.11 and compiled it with the three proxy
enablers /--enable-proxy, --enable-proxy-balancer, and --enable-proxy-
http/. I copied the reverse proxy config part and the JkMount
directives from the VmWare virtual machine into the httpd.conf file on
the Mac.
Now I have two machines:
- A PC with Windows 2000 and a Mac with OSX 10.5.6. The PC has
apache2 + mod_jk + tomcat (1 instance).
- A Mac that has apache2 + mod_jk + tomcat (3 instances). The Mac
also acts as a reverse proxy.
Fired up the newly compiled apache2 on the Mac and now everything
works like charm. It is failing over nicely when I stop the selected
tomcat node. I still have to test the fail over when I stop apache on
the PC in the middle of a session.
I changed two things:
1. moved the reverse proxy configuration off from the Windows Xp
inside the VmWare Fussion machine on the Mac, so the Mac itself became
the reverse proxy. Nothing was done to the tomcats or to the apache2
on the PC, not even stopping them.
2. On the Mac I installed a newer version of apache2. /2.2.11
compared to 2.2.9/
My guess is the VmWare update and the WinXp update on the virtual
machine from December changed somehow the security settings that did
not allow the information to flow back to the reverse proxy. Of
course, I am not sure about it. :) Theoretically it should have
worked without any problems. If I have time I will install a 2.2.11
apache2 on the XP and move back to it the reverse proxy configuration
from the Mac, to see if that helps or not.
Thanks ahead,
János
On Mar 25, 2009, at 6:49 PM, Rainer Jung wrote:
> On 20.03.2009 18:48, János Löbb wrote:
>> Hi,
>>
>> I have two real machines. One of them is a Windows XP running Apache
>> 2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The
>> other
>> one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26 and
>> Tomcat 6.0.16.
>>
>> The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, -
>> node1,2,3. Altogether 4 tomcats and they form a cluster.
>
> ...
>
>> In front of these two machines is an Apache 2.2.10 in a VMWare Fusion
>> virtual Machine on Windows XP in the Mac and configured as a
>> Reverse proxy
>>
>> Here is the reverse proxy config from the httpd.conf
>>
>> <snip>
>> ProxyRequests Off
>> <Proxy balancer://pathCluster>
>> BalancerMember http://bml0039.yalepath.org loadfactor=10
>> BalancerMember http://bml0073.yalepath.org loadfactor=10
>> ProxySet lbmethod=bytraffic
>> </Proxy>
>> ProxyPass /tc/ balancer://pathCluster/
>> ProxyPassReverse /tc/ balancer://pathCluster/
>>
>>
>> <Location /balancer-manager>
>> SetHandler balancer-manager
>> Order Deny,Allow
>> Allow from .yalepath.org
>> </Location>
>> </snip>
>
>> Here is the access_log from the reverse proxy:
>>
>> <snip>
>> 10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] "GET /tc/CassetteLabeler
>> HTTP/1.1" 302 -
>
> Returns with a trailing slash redirect
>
>> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /tc/
>> CassetteLabeler/
>> HTTP/1.1" 200 1506
>
> Returns Content with 1506 Bytes
>
>> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET
>> /CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html
>> HTTP/1.1" 404 260
>
> Wrong request, Prefix /tc/ missing. You need to find out, where this
> request comes from. Likely it is either generated by someone using a
> link in the page returned by the previous request, or that page
> contains a meta tag in the head, which triggers a redirect to this
> wrong URL.
>
> So retrieve /tc/CassetteLabeler/ and have a look at the content to
> identify, whether the wrong URL "/CassetteLabeler/
> pathology.labeler.labeler.Labeler/Labeler.html" is in the page.
>
> If no, we have to think further.
> If yes, you have to ask, why is this wrong URL in there. The
> question is, how your application generates this URL. If it
> generates absolute URLs instead of relative ones, then you can
> easily run into trouble, when the path of the real application is
> different from the path used from the outside.
>
> It is not unlikely though, that in such a case the app or the
> framework used has a configurable path prefix. But first check, if
> you can verify the wrong URL is somewhere in the content of the page.
>
>
>> The request from the reverse proxy did go to the XP machine:
>> Here are the log entries from the access log:
>>
>> <snip>
>> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler
>> HTTP/1.1" 302 -
>
> That's only the first request, answered with the trailing slash
> redirect. The next request is the interesting one (the one answered
> with status 200). It also has to appear somewhere in your 2nd laqyer
> httpd logs.
>
>> Here are the entries from mod_jk.log of this XP machine:
>>
>> <snip>
>> [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c
>> (682): Attempting to map URI '/CassetteLabeler' from 5 maps
>
> Again that's the first request
>
>
>> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
>> (608):
>> status = 302
>> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
>> (615):
>> Number of headers is = 1
>> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c
>> (671):
>> Header[0] [Location] = [http://bml0039.yalepath.org/CassetteLabeler/]
>
> Aha, we get back a redirect (302) to localtion http://bml0039.yalepath.org/CassetteLabeler/
> (which will be further rewritten by ProxyPassReverse). But that we
> already know. The next request is the interesting one.
>
>
>> I have a production environment - just with 3 Macs and just one
>> tomcat
>> per machine forming a cluster - where a very similar configuration
>> runs
>> fine. The only difference is that the sticky parameters are NOT
>> with the
>> load balancer but with the real workers in the workers.properties
>> file,
>
> This is very likely not the cause.
>
>> and the reverse proxy configuration somewhat different. Here it is:
>>
>> <snip>
>> <IfModule mod_proxy_balancer.c>
>> ProxyPass /apps/ balancer://balancer-group/ stickysession=JSESSIONID
>> ProxyPassReverse /apps/ balancer://balancer-group/
>> <Proxy "balancer://balancer-group">
>> BalancerMember http://pathsrv1.yalepath.org loadfactor=10
>> BalancerMember http://raid2b.yalepath.org loadfactor=10
>> BalancerMember http://reptile.yalepath.org loadfactor=10
>> </Proxy>
>> </IfModule>
>
> Looks pretty analogous at first sight.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: reversed proxy stopped working with tomcat cluster
Posted by Rainer Jung <ra...@kippdata.de>.
On 20.03.2009 18:48, János Löbb wrote:
> Hi,
>
> I have two real machines. One of them is a Windows XP running Apache
> 2.2.10 + mod_jk /release date of 10/30/2008/ + Tomcat 6.0.16. The other
> one is a Mac with OSX 10.5.6 with Apache 2.2.9 + mod_jk 1.2.26 and
> Tomcat 6.0.16.
>
> The XP machine runs one instance of Tomcat /node4/. The Mac runs 3, -
> node1,2,3. Altogether 4 tomcats and they form a cluster.
...
> In front of these two machines is an Apache 2.2.10 in a VMWare Fusion
> virtual Machine on Windows XP in the Mac and configured as a Reverse proxy
>
> Here is the reverse proxy config from the httpd.conf
>
> <snip>
> ProxyRequests Off
> <Proxy balancer://pathCluster>
> BalancerMember http://bml0039.yalepath.org loadfactor=10
> BalancerMember http://bml0073.yalepath.org loadfactor=10
> ProxySet lbmethod=bytraffic
> </Proxy>
> ProxyPass /tc/ balancer://pathCluster/
> ProxyPassReverse /tc/ balancer://pathCluster/
>
>
> <Location /balancer-manager>
> SetHandler balancer-manager
> Order Deny,Allow
> Allow from .yalepath.org
> </Location>
> </snip>
> Here is the access_log from the reverse proxy:
>
> <snip>
> 10.84.2.195 - - [20/Mar/2009:12:09:48 -0400] "GET /tc/CassetteLabeler
> HTTP/1.1" 302 -
Returns with a trailing slash redirect
> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /tc/CassetteLabeler/
> HTTP/1.1" 200 1506
Returns Content with 1506 Bytes
> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET
> /CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html
> HTTP/1.1" 404 260
Wrong request, Prefix /tc/ missing. You need to find out, where this
request comes from. Likely it is either generated by someone using a
link in the page returned by the previous request, or that page contains
a meta tag in the head, which triggers a redirect to this wrong URL.
So retrieve /tc/CassetteLabeler/ and have a look at the content to
identify, whether the wrong URL
"/CassetteLabeler/pathology.labeler.labeler.Labeler/Labeler.html" is in
the page.
If no, we have to think further.
If yes, you have to ask, why is this wrong URL in there. The question
is, how your application generates this URL. If it generates absolute
URLs instead of relative ones, then you can easily run into trouble,
when the path of the real application is different from the path used
from the outside.
It is not unlikely though, that in such a case the app or the framework
used has a configurable path prefix. But first check, if you can verify
the wrong URL is somewhere in the content of the page.
> The request from the reverse proxy did go to the XP machine:
> Here are the log entries from the access log:
>
> <snip>
> 10.84.2.195 - - [20/Mar/2009:12:09:49 -0400] "GET /CassetteLabeler
> HTTP/1.1" 302 -
That's only the first request, answered with the trailing slash
redirect. The next request is the interesting one (the one answered with
status 200). It also has to appear somewhere in your 2nd laqyer httpd logs.
> Here are the entries from mod_jk.log of this XP machine:
>
> <snip>
> [Fri Mar 20 12:09:49.254 2009] [852:1808] [debug] jk_uri_worker_map.c
> (682): Attempting to map URI '/CassetteLabeler' from 5 maps
Again that's the first request
> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (608):
> status = 302
> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (615):
> Number of headers is = 1
> [Fri Mar 20 12:09:49.551 2009] [852:1808] [debug] jk_ajp_common.c (671):
> Header[0] [Location] = [http://bml0039.yalepath.org/CassetteLabeler/]
Aha, we get back a redirect (302) to localtion
http://bml0039.yalepath.org/CassetteLabeler/ (which will be further
rewritten by ProxyPassReverse). But that we already know. The next
request is the interesting one.
> I have a production environment - just with 3 Macs and just one tomcat
> per machine forming a cluster - where a very similar configuration runs
> fine. The only difference is that the sticky parameters are NOT with the
> load balancer but with the real workers in the workers.properties file,
This is very likely not the cause.
> and the reverse proxy configuration somewhat different. Here it is:
>
> <snip>
> <IfModule mod_proxy_balancer.c>
> ProxyPass /apps/ balancer://balancer-group/ stickysession=JSESSIONID
> ProxyPassReverse /apps/ balancer://balancer-group/
> <Proxy "balancer://balancer-group">
> BalancerMember http://pathsrv1.yalepath.org loadfactor=10
> BalancerMember http://raid2b.yalepath.org loadfactor=10
> BalancerMember http://reptile.yalepath.org loadfactor=10
> </Proxy>
> </IfModule>
Looks pretty analogous at first sight.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org