Seeking tips on multi-host, multi-context deployments

[Sam Hokin <sa...@ims.net>]

Hi, folks.  I've got what I consider to be a fairly significant production implementation of Tomcat with, currently, 89 
contexts in 30 hosts on two concurrent servers.  Everything works fine ... except when it doesn't.  As the number of 
contexts and hosts have increased, there have been more and more JVM and/or Tomcat crashes, typically a couple per week 
lately, on one or both servers.  The error messages vary from nothing at all in catalina.out, to exceptions regarding 
database connections, to exceptions about threads.  Basically, I've been trying to find some good advice on 
configuration in my situation and haven't been able to find much.

In a bit more detail, the deployment is as follows:

(2) Dell 1950 8-CPU servers running Fedora 14 Linux 2.6.35, one with 2 GB RAM and one with 4 GB RAM
(2) Tomcat 7.0.16 instances, one on each server, running clustered contexts in "distributable" mode, under Sun JDK 1.6.0
One server runs Apache HTTPD, using mod_proxy_ajp to connect to the two Tomcat instances and manage balancing
The other server runs PostgreSQL, the database used by the apps, which connect to it via JNDI/JDBC connection pooling

There are a few contexts that run on only one or the other server (namely, Softslate and Searchblox, which are not 
serialized to support clustering).   Most contexts, which run my app, run in parallel on both servers.  I've implemented 
clustering to maintain site availability when a Tomcat instance goes down.

I could post up all the various JVM and Tomcat parameters, but I'm really looking for any definitive list of tips for 
things to try to improve stability.  If anyone has written or seen a "how-to" document on multi-host, multi-context 
deployments like this, please share!  Thanks!!

P.S. One option I've considered is to run KVM and run each host in its own virtual OS.  To me that's a ridiculous amount 
of OS overhead to deal with occasional Tomcat outages, but it probably would be one solution, much more difficult to 
maintain than a single Tomcat instance per server running all the hosts and contexts.  If you've had experience running 
multiple Tomcat instances in separate virtual machines, I'd be curious to hear how it went.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Seeking tips on multi-host, multi-context deployments

[David Kerber <dc...@verizon.net>]

On 9/28/2011 7:25 PM, Sam Hokin wrote:
> Hi, folks.  I've got what I consider to be a fairly significant 
> production implementation of Tomcat with, currently, 89 contexts in 30 
> hosts on two concurrent servers.  Everything works fine ... except 
> when it doesn't.  As the number of contexts and hosts have increased, 
> there have been more and more JVM and/or Tomcat crashes, typically a 
> couple per week lately, on one or both servers.  The error messages 
> vary from nothing at all in catalina.out, to exceptions regarding 
> database connections, to exceptions about threads.  Basically, I've 
> been trying to find some good advice on configuration in my situation 
> and haven't been able to find much.
>
> In a bit more detail, the deployment is as follows:
>
> (2) Dell 1950 8-CPU servers running Fedora 14 Linux 2.6.35, one with 2 
> GB RAM and one with 4 GB RAM
Stuff in lots more RAM!


> (2) Tomcat 7.0.16 instances, one on each server, running clustered 
> contexts in "distributable" mode, under Sun JDK 1.6.0
> One server runs Apache HTTPD, using mod_proxy_ajp to connect to the 
> two Tomcat instances and manage balancing
> The other server runs PostgreSQL, the database used by the apps, which 
> connect to it via JNDI/JDBC connection pooling
>
> There are a few contexts that run on only one or the other server 
> (namely, Softslate and Searchblox, which are not serialized to support 
> clustering).   Most contexts, which run my app, run in parallel on 
> both servers.  I've implemented clustering to maintain site 
> availability when a Tomcat instance goes down.
>
> I could post up all the various JVM and Tomcat parameters, but I'm 
> really looking for any definitive list of tips for things to try to 
> improve stability.  If anyone has written or seen a "how-to" document 
> on multi-host, multi-context deployments like this, please share!  
> Thanks!!
>
> P.S. One option I've considered is to run KVM and run each host in its 
> own virtual OS.  To me that's a ridiculous amount of OS overhead to 
> deal with occasional Tomcat outages, but it probably would be one 
> solution, much more difficult to maintain than a single Tomcat 
> instance per server running all the hosts and contexts.  If you've had 
> experience running multiple Tomcat instances in separate virtual 
> machines, I'd be curious to hear how it went.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Seeking tips on multi-host, multi-context deployments

[Mark Thomas <ma...@apache.org>]

On 28/09/2011 19:25, Sam Hokin wrote:
> (2) Dell 1950 8-CPU servers running Fedora 14 Linux 2.6.35, one with 2
> GB RAM and one with 4 GB RAM

That seems low but if it works for you...

> (2) Tomcat 7.0.16 instances, one on each server, running clustered
> contexts in "distributable" mode, under Sun JDK 1.6.0

Worth updating to 7.0.21 (or 7.0.22 which will hopefully be released in
a few days).

> P.S. One option I've considered is to run KVM and run each host in its
> own virtual OS.  To me that's a ridiculous amount of OS overhead to deal
> with occasional Tomcat outages, but it probably would be one solution,
> much more difficult to maintain than a single Tomcat instance per server
> running all the hosts and contexts.  If you've had experience running
> multiple Tomcat instances in separate virtual machines, I'd be curious
> to hear how it went.

I wouldn't go as far as a VM per host but you might want to consider
multiple Tomcat instances per server with, for example, a Tomcat
instance per host. Separate CATALINA_HOME (the binary) and CATALINA_BASE
(one per host) provides separate Java process for each host with minimal
overhead. Your memory requirements will increase but not by much. You
can split the apps between hosts any way you like.

The ASF uses this to run the main, test1 and test2 Jira instances from a
single Tomcat binary. Tomcat upgrades take me about 5 minutes from start
to finish with about 2 minutes downtime. The downtime is pretty much all
due to the time it takes Jira to stop and start.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Seeking tips on multi-host, multi-context deployments

[Igor Cicimov <ic...@gmail.com>]

If i was you i wouldnt even start without 8GB of RAM on each server!
 On Sep 29, 2011 9:26 AM, "Sam Hokin" <sa...@ims.net> wrote:
> Hi, folks. I've got what I consider to be a fairly significant production
implementation of Tomcat with, currently, 89
> contexts in 30 hosts on two concurrent servers. Everything works fine ...
except when it doesn't. As the number of
> contexts and hosts have increased, there have been more and more JVM
and/or Tomcat crashes, typically a couple per week
> lately, on one or both servers. The error messages vary from nothing at
all in catalina.out, to exceptions regarding
> database connections, to exceptions about threads. Basically, I've been
trying to find some good advice on
> configuration in my situation and haven't been able to find much.
>
> In a bit more detail, the deployment is as follows:
>
> (2) Dell 1950 8-CPU servers running Fedora 14 Linux 2.6.35, one with 2 GB
RAM and one with 4 GB RAM
> (2) Tomcat 7.0.16 instances, one on each server, running clustered
contexts in "distributable" mode, under Sun JDK 1.6.0
> One server runs Apache HTTPD, using mod_proxy_ajp to connect to the two
Tomcat instances and manage balancing
> The other server runs PostgreSQL, the database used by the apps, which
connect to it via JNDI/JDBC connection pooling
>
> There are a few contexts that run on only one or the other server (namely,
Softslate and Searchblox, which are not
> serialized to support clustering). Most contexts, which run my app, run in
parallel on both servers. I've implemented
> clustering to maintain site availability when a Tomcat instance goes down.
>
> I could post up all the various JVM and Tomcat parameters, but I'm really
looking for any definitive list of tips for
> things to try to improve stability. If anyone has written or seen a
"how-to" document on multi-host, multi-context
> deployments like this, please share! Thanks!!
>
> P.S. One option I've considered is to run KVM and run each host in its own
virtual OS. To me that's a ridiculous amount
> of OS overhead to deal with occasional Tomcat outages, but it probably
would be one solution, much more difficult to
> maintain than a single Tomcat instance per server running all the hosts
and contexts. If you've had experience running
> multiple Tomcat instances in separate virtual machines, I'd be curious to
hear how it went.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

Re: Seeking tips on multi-host, multi-context deployments

[Sam Hokin <sa...@ims.net>]

HUGE apologies, Pid! I had no idea that a changed subject would keep my email in a previous thread.  I saw no visible 
connection to your thread, I didn't know that my email client (Thunderbird) would pass the reference to your thread back 
in the email header.  I'll be sure to start a fresh thread in the future, very embarrassing.  Believe me, no thread 
hijacking was intended, it was just mailing list incompetence.  (I also have no idea how to pull that reference from 
this reply, sorry.)

And thanks for the tip, Mark, I'll give multiple Tomcat instances a try, I guess that's pretty obvious, I was trying to 
reduce the overhead of updates by running a single container.  And I thought that having 30-odd Tomcat instances (or 
whatever) would be a much larger drain on resources than a single Tomcat instance with 30 hosts; sounds like it isn't.

And more memory - yes.  I'll get those servers up to 8 GB each.  But what I was looking for was a reference to some sort 
of multi-context multi-host "HOW-TO", which I'm guessing does not exist.  I don't want to ask you folks to get into the 
details of my specific situation, that's my job.

On 9/29/2011 1:19 AM, Pid wrote:
> All of which is splendid, but it would've been better if you hadn't
> hijacked the "Incorporating changes&  compiling Tomcat" thread by
> replying to my last message&  editing the subject&  body.
>
> Please start a new thread instead, in future.
>
>
> p

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Seeking tips on multi-host, multi-context deployments

[Pid <pi...@pidster.com>]

All of which is splendid, but it would've been better if you hadn't
hijacked the "Incorporating changes & compiling Tomcat" thread by
replying to my last message & editing the subject & body.

Please start a new thread instead, in future.


p

RE: Seeking tips on multi-host, multi-context deployments

["Caldarale, Charles R" <Ch...@unisys.com>]

> From: Sam Hokin [mailto:sam@ims.net] 
> Subject: Seeking tips on multi-host, multi-context deployments

> I could post up all the various JVM and Tomcat parameters

The devil is in the details.  Without the specifics of the execution environment and the observed errors, it's really impossible to give you anything definitive.  More RAM is often useful, but frequently only postpones outages due to ill-behaved applications (the most common cause of crashes).

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.