You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Decker, Richard M" <Ri...@uth.tmc.edu> on 2016/05/05 19:29:30 UTC
BackupManager Heterogeneous Environment / Functionality?
ASF Tomcat Members,
I'm trying to configure & demo an Apache Tomcat [v8] cluster env, that supports different apps functioning on different systems. Basically, I'm trying to setup an environment where an app can be on only 1 or 2 (deployed or running) out of the 3 Tomcat systems in the cluster. However, this does not seem to work for me, as I can't get away from the 404s when Apache HTTPD goes to a random (round robin) Tomcat system, that does not have the application (running or deployed) on it. This happens with both new & cached sessions. I can bring an entire Tomcat system down, and it works as designed, but not for (stopped or undeployed) single applications. I'm really not sure how Tomcat and Apache are supposed to communicate with each other in this regard; so they know what apps are deployed/running on each system. Is this even possible? According to the link below, it should work...
http://tomcat.apache.org/tomcat-8.0-doc/config/cluster-manager.html
The org.apache.catalina.ha.session.BackupManager also replicates deltas but only to one backup node. The location of the backup node is known to all nodes in the cluster. It also supports heterogeneous deployments, so the manager knows at what locations the web application is deployed.
--------------------
Env - Top Down
--------------------
[Load Balancer] - (Sticky Sessions)
|
[HTTPD1] - [HTTPD2] - (Load Balanced, Sticky Sessions)
|
[Tomcat1] - [Tomcat2] - [Tomcat3] - (BackupManager)
------
Stats
------
[HTTPD] X 2
Server version: Apache/2.4.18 (Unix)
Server built: Dec 21 2015 14:42:09
[Tomcat] X 3
Server version: Apache Tomcat/8.0.33
Server built: Mar 18 2016 20:31:49
Server number: 8.0.33.0
OS Name: Linux
OS Version: 3.10.0-327.4.4.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_77-b03
JVM Vendor: Oracle Corporation
[Linux] X 3 Tomcat Systems
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 all-systems.mcast.net
eth0 1 228.0.0.4
eth0 1 all-systems.mcast.net
eth1 1 all-systems.mcast.net
--------
Config
--------
[HTTPD]
worker.list=tomcattest, statustest
worker.tomcat1.type=ajp13
worker.tomcat1.host=tomcat1
worker.tomcat1.port=XXXX
#worker.tomcat1.lbfactor=5
worker.tomcat2.type=ajp13
worker.tomcat2.host=tomcat2
worker.tomcat2.port=XXXX
#worker.tomcat2.lbfactor=5
worker.tomcat3.type=ajp13
worker.tomcat3.host=tomcat3
worker.tomcat3.port=XXXX
#worker.tomcat3.lbfactor=5
worker.tomcattest.type=lb
worker.tomcattest.balance_workers=tomcat1,tomcat2,tomcat3
worker.tomcattest.sticky_session=True
# lb methods: [R]equest, [S]ession, [T]raffic, [B]usiness
worker.tomcattest.method=R
worker.statustest.type=status
[tomcat] X3 - server.xml
<!-- Defining the AJP 1.3 Connector on Port XXXX -->
<Connector port="XXXX" protocol="AJP/1.3" redirectPort="XXXX" />
<!-- Defining the Local Host & Route to the local machine -->
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcat1/2/3">
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.BackupManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"
mapSendOptions="6"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.4"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="4000"
autoBind="100"
selectorTimeout="5000"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
-----------------
Cluster Basics
-----------------
All your session attributes must implement java.io.Serializable - Done
Uncomment the Cluster element in server.xml - Done
If you have defined custom cluster valves, make sure you have the ReplicationValve defined as well under the Cluster element in server.xml - N/A
If your Tomcat instances are running on the same machine, make sure the Receiver.port attribute is unique for each instance, in most cases Tomcat is smart enough to resolve this on it's own by autodetecting available ports in the range 4000-4100 - N/A
Make sure your web.xml has the <distributable/> element - Done
If you are using mod_jk, make sure that jvmRoute attribute is set at your Engine <Engine name="Catalina" jvmRoute="node01" > and that the jvmRoute attribute value matches your worker name in workers.properties - Done
Make sure that all nodes have the same time and sync with NTP service! - Done
Make sure that your loadbalancer is configured for sticky session mode - Done
Thanks for reading!
Re: BackupManager Heterogeneous Environment / Functionality?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard,
On 5/5/16 6:01 PM, Decker, Richard M wrote:
> Thanks for the quick reply. The info you requested is below. The
> config is identical (minus machine names) on the 2 apache httpd
> systems.
>
> [httpd.conf] equivalent - All JkUn/Mounts
>
> <virtualhost *XXX> # All requests go to tomcattest by default
> JkMount /* acttomcattest # examples: # Static files in the examples
> webapp are served by apache #Alias /examples
> /tomcat/webapps/examples # here we serve the generic default page
> via apache instead of tomcat JkUnMount / tomcattest JkUnMount
> /index.html tomcattest JkUnMount /logo.gif tomcattest # Serve gif
> using httpd #JkUnMount /*.gif tomcattest
>
> <virtualhost *XXX> JkMount /status statustest
So you are JkMounting everything to a single, large load-balancer
worker. I would recommend a per-application load-balancer-worker
setup, but for many applications, that can indeed become tedious. You
can get around that with another layer of indirection, I suspect.
>>> This happens with both new & cached sessions. I can bring an
>>> entire Tomcat system down, and it works as designed, but not
>>> for (stopped or undeployed) single applications. I'm really not
>>> sure how Tomcat and Apache are supposed to communicate with
>>> each other in this regard; so they know what apps are
>>> deployed/running on each system. Is this even possible?
>>
>> Yep. You just need to tell your load-balancer about which
>> applications can be found where.
>
> So you're suggesting that I list every app in the apache/httpd
> config, or just those that won't be deployed across the whole (3
> system) env? Sound promising , but problematic of having to
> restart the Apache service, whenever a change is made.
I suppose only those you want to place on a limited number of servers.
But you'll always have the problem that any of the "globally-deployed"
applications being taken-down for maintenance will return 404 errors
from the nodes on which they run. It's very helpful to be able to
disable a single node/application combination, and you can't really do
that unless your load-balancer knows about each individual application.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlcw9S4ACgkQ9CaO5/Lv0PCD5ACghLNwXpBryoxB2L8fs4N9imVx
uRsAn0wNdWk7hDF2rcesql5DTaZ3LGXV
=jxjf
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: BackupManager Heterogeneous Environment / Functionality?
Posted by "Decker, Richard M" <Ri...@uth.tmc.edu>.
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Thursday, May 05, 2016 4:26 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: BackupManager Heterogeneous Environment / Functionality?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard,
On 5/5/16 3:29 PM, Decker, Richard M wrote:
> I'm trying to configure & demo an Apache Tomcat [v8] cluster env, that
> supports different apps functioning on different systems.
>
> Basically, I'm trying to setup an environment where an app can be on
> only 1 or 2 (deployed or running) out of the 3 Tomcat systems in the
> cluster. However, this does not seem to work for me, as I can't get
> away from the 404s when Apache HTTPD goes to a random (round robin)
> Tomcat system, that does not have the application (running or
> deployed) on it.
The one thing you forgot to post was your JkMounts from httpd.conf (or similar). Can you post those? I believe this can be entirely solved with a slightly more complicated configuration.
Chris,
Thanks for the quick reply. The info you requested is below. The config is identical (minus machine names) on the 2 apache httpd systems.
[httpd.conf] equivalent - All JkUn/Mounts
<virtualhost *XXX>
# All requests go to tomcattest by default
JkMount /* acttomcattest
# examples:
# Static files in the examples webapp are served by apache
#Alias /examples /tomcat/webapps/examples
# here we serve the generic default page via apache instead of tomcat
JkUnMount / tomcattest
JkUnMount /index.html tomcattest
JkUnMount /logo.gif tomcattest
# Serve gif using httpd
#JkUnMount /*.gif tomcattest
<virtualhost *XXX>
JkMount /status statustest
> This happens with both new & cached sessions. I can bring an entire
> Tomcat system down, and it works as designed, but not for (stopped or
> undeployed) single applications. I'm really not sure how Tomcat and
> Apache are supposed to communicate with each other in this regard; so
> they know what apps are deployed/running on each system. Is this even
> possible?
Yep. You just need to tell your load-balancer about which applications can be found where.
So you're suggesting that I list every app in the apache/httpd config, or just those that won't be deployed across the whole (3 system) env? Sound promising , but problematic of having to restart the Apache service, whenever a change is made.
> According to the link below, it should work...
>
> http://tomcat.apache.org/tomcat-8.0-doc/config/cluster-manager.html
>
>
>
>
The org.apache.catalina.ha.session.BackupManager also replicates
> deltas but only to one backup node. The location of the backup node is
> known to all nodes in the cluster. It also supports heterogeneous
> deployments, so the manager knows at what locations the web
> application is deployed.
>
> -------------------- Env - Top Down --------------------
>
> [Load Balancer] - (Sticky Sessions) | [HTTPD1] - [HTTPD2] - (Load
> Balanced, Sticky Sessions) | [Tomcat1] - [Tomcat2] - [Tomcat3] -
> (BackupManager)
Okay.
> -------- Config --------
>
> [HTTPD] worker.list=tomcattest, statustest
>
> worker.tomcat1.type=ajp13 worker.tomcat1.host=tomcat1
> worker.tomcat1.port=XXXX #worker.tomcat1.lbfactor=5
>
> worker.tomcat2.type=ajp13 worker.tomcat2.host=tomcat2
> worker.tomcat2.port=XXXX #worker.tomcat2.lbfactor=5
>
> worker.tomcat3.type=ajp13 worker.tomcat3.host=tomcat3
> worker.tomcat3.port=XXXX #worker.tomcat3.lbfactor=5
>
> worker.tomcattest.type=lb
> worker.tomcattest.balance_workers=tomcat1,tomcat2,tomcat3
> worker.tomcattest.sticky_session=True # lb methods: [R]equest,
> [S]ession, [T]raffic, [B]usiness worker.tomcattest.method=R
> worker.statustest.type=status
If you define more than one lb worker, you can do this in a more nuanced way. For example:
worker.tomcat1.host=tomcat1
worker.tomcat1.port=XXXX
worker.tomcat2.host=tomcat2
worker.tomcat2.port=XXXX
worker.tomcat3.host=tomcat3
worker.tomcat3.port=XXXX
worker.app-a.type=lb
worker.app-a.balance_workers=tomcat1,tomcat2
worker.app-a.sticky_session=True
worker.app-a.type=lb
worker.app-a.balance_workers=tomcat2,tomcat3
worker.app-a.sticky_session=True
Then, in httpd.conf:
JkMount /app-a/* app-a
JkMount /app-b/* app-b
I think I follow you here.
You can just deploy the applications wherever you want them.
Now, how do you deploy them everywhere but then allow one of them to be taken out of service without taking-down the whole Tomcat instance?
Tomcat app manager - stopping the application. Or removing/deleting/ undeploying the application itself while Tomcat is (hot) running. This will most likely break the config for Apache <-> Tomcat though?
There are several ways of doing that which I'll summarize, here. Let's start a new thread if you want to ask about how to do this in further detail.
1. Deploy a ROOT application on each Tomcat node that will return a "failure" status whenever you request /app-a/* or /app-b/* (etc.).
Maybe the ROOT web application ALWAYS returns an error status.
2. Use the mod_jk status worker to set the activation status of one of the Tomcat nodes to "DIS" (disabled). This will send requests without a session identifier to other nodes in the balancer. Set it to "STO"
and it will cause *all* requests to be sent to other nodes.
The first method is more fault-tolerant and lazy, since you don't have to tell mod_jk when nodes will be going down. The second method is a little more work, but it also means that you can drain nodes before scheduled maintenance, which is nice.
Interesting, I will need to research both of those suggestions a bit more, to understand them better.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlcrukwACgkQ9CaO5/Lv0PA1QwCgroEfkUXGfhJnGVWu4PlWES4v
mqAAnA3+F9TGgC1jLbGD7SOh5/4JkJAS
=KvPU
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BackupManager Heterogeneous Environment / Functionality?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Richard,
On 5/5/16 3:29 PM, Decker, Richard M wrote:
> I'm trying to configure & demo an Apache Tomcat [v8] cluster env,
> that supports different apps functioning on different systems.
>
> Basically, I'm trying to setup an environment where an app can be
> on only 1 or 2 (deployed or running) out of the 3 Tomcat systems
> in the cluster. However, this does not seem to work for me, as I
> can't get away from the 404s when Apache HTTPD goes to a random
> (round robin) Tomcat system, that does not have the application
> (running or deployed) on it.
The one thing you forgot to post was your JkMounts from httpd.conf (or
similar). Can you post those? I believe this can be entirely solved
with a slightly more complicated configuration.
> This happens with both new & cached sessions. I can bring an
> entire Tomcat system down, and it works as designed, but not for
> (stopped or undeployed) single applications. I'm really not sure
> how Tomcat and Apache are supposed to communicate with each other
> in this regard; so they know what apps are deployed/running on each
> system. Is this even possible?
Yep. You just need to tell your load-balancer about which applications
can be found where.
> According to the link below, it should work...
>
> http://tomcat.apache.org/tomcat-8.0-doc/config/cluster-manager.html
>
>
>
>
The org.apache.catalina.ha.session.BackupManager also replicates
> deltas but only to one backup node. The location of the backup
> node is known to all nodes in the cluster. It also supports
> heterogeneous deployments, so the manager knows at what locations
> the web application is deployed.
>
> -------------------- Env - Top Down --------------------
>
> [Load Balancer] - (Sticky Sessions) | [HTTPD1] - [HTTPD2] - (Load
> Balanced, Sticky Sessions) | [Tomcat1] - [Tomcat2] - [Tomcat3] -
> (BackupManager)
Okay.
> -------- Config --------
>
> [HTTPD] worker.list=tomcattest, statustest
>
> worker.tomcat1.type=ajp13 worker.tomcat1.host=tomcat1
> worker.tomcat1.port=XXXX #worker.tomcat1.lbfactor=5
>
> worker.tomcat2.type=ajp13 worker.tomcat2.host=tomcat2
> worker.tomcat2.port=XXXX #worker.tomcat2.lbfactor=5
>
> worker.tomcat3.type=ajp13 worker.tomcat3.host=tomcat3
> worker.tomcat3.port=XXXX #worker.tomcat3.lbfactor=5
>
> worker.tomcattest.type=lb
> worker.tomcattest.balance_workers=tomcat1,tomcat2,tomcat3
> worker.tomcattest.sticky_session=True # lb methods: [R]equest,
> [S]ession, [T]raffic, [B]usiness worker.tomcattest.method=R
> worker.statustest.type=status
If you define more than one lb worker, you can do this in a more
nuanced way. For example:
worker.tomcat1.host=tomcat1
worker.tomcat1.port=XXXX
worker.tomcat2.host=tomcat2
worker.tomcat2.port=XXXX
worker.tomcat3.host=tomcat3
worker.tomcat3.port=XXXX
worker.app-a.type=lb
worker.app-a.balance_workers=tomcat1,tomcat2
worker.app-a.sticky_session=True
worker.app-a.type=lb
worker.app-a.balance_workers=tomcat2,tomcat3
worker.app-a.sticky_session=True
Then, in httpd.conf:
JkMount /app-a/* app-a
JkMount /app-b/* app-b
You can just deploy the applications wherever you want them.
Now, how do you deploy them everywhere but then allow one of them to
be taken out of service without taking-down the whole Tomcat instance?
There are several ways of doing that which I'll summarize, here. Let's
start a new thread if you want to ask about how to do this in further
detail.
1. Deploy a ROOT application on each Tomcat node that will return a
"failure" status whenever you request /app-a/* or /app-b/* (etc.).
Maybe the ROOT web application ALWAYS returns an error status.
2. Use the mod_jk status worker to set the activation status of one of
the Tomcat nodes to "DIS" (disabled). This will send requests without
a session identifier to other nodes in the balancer. Set it to "STO"
and it will cause *all* requests to be sent to other nodes.
The first method is more fault-tolerant and lazy, since you don't have
to tell mod_jk when nodes will be going down. The second method is a
little more work, but it also means that you can drain nodes before
scheduled maintenance, which is nice.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlcrukwACgkQ9CaO5/Lv0PA1QwCgroEfkUXGfhJnGVWu4PlWES4v
mqAAnA3+F9TGgC1jLbGD7SOh5/4JkJAS
=KvPU
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org