You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Baldur <ba...@gmail.com> on 2014/04/28 14:00:24 UTC
Tomcat configuration with multiple webapps
Hi all,
I'd like to get some help about my current architecture. The
current scenario uses mod_jk to connect Apache httpd and Tomcat6. I have two
Tomcat instances (using DeltaManager for session replication and sticky
session enabled) in order to provide high availability and balance load
across instances. Currently Tomcat manages 28 webapps and 7 of them are
only web services. Generally speaking, a webapp usually involves JSF or
Struts while a web services war involves JAX-WS. Both types of application
have a common stack implemented with Spring and Hibernate. As a result, each
application produces a war of around 40-50 MB.
I'd like to ask you several questions to provide better
performance:
* Which approach would be appropriate for this scenario? All wars in
one cluster? Maybe move web services to other cluster?
* In order to improve deployments, which technique can I use to
minimize war size? Will be the cause of memory issues? I have tried to put
some common jars (spring, apache-commons and so on) in Tomcat lib but I
don't know if there is a better approach by other means.
I read as much as I can but I'm stuck trying to find the best tools to
monitor the system and tackle memory issues (such as the dreaded PermGen).
I think it's a quite common scenario for a relatively small production
environment but I don't find the best configuration that suits this type of
deployment.
Any help would be much appreciated.
Thanks very much in advance
Cheers
Re: Tomcat configuration with multiple webapps
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Baldur,
On 4/28/14, 8:00 AM, Baldur wrote:
> Hi all,
>
> I'd like to get some help about my current architecture. The
> current scenario uses mod_jk to connect Apache httpd and Tomcat6. I
> have two Tomcat instances (using DeltaManager for session
> replication and sticky session enabled) in order to provide high
> availability and balance load across instances. Currently Tomcat
> manages 28 webapps and 7 of them are only web services. Generally
> speaking, a webapp usually involves JSF or Struts while a web
> services war involves JAX-WS. Both types of application have a
> common stack implemented with Spring and Hibernate. As a result,
> each application produces a war of around 40-50 MB.
>
> I'd like to ask you several questions to provide better
> performance:
>
> * Which approach would be appropriate for this scenario?
> All wars in one cluster? Maybe move web services to other cluster?
Honestly, this is going to depend more on your own application's
architecture, your own deployment preferences, etc. One can generally
make a good argument about just about any strategy, here.
> * In order to improve deployments, which technique can I
> use to minimize war size? Will be the cause of memory issues? I
> have tried to put some common jars (spring, apache-commons and so
> on) in Tomcat lib but I don't know if there is a better approach by
> other means.
What is your motivation to reduce WAR size? 50MiB is practically nothing.
> I read as much as I can but I'm stuck trying to find the best tools
> to monitor the system and tackle memory issues (such as the dreaded
> PermGen).
Are you actually experiencing PermGen exhaustion? What steps have you
already taken? I don't use Struts 2 or Spring, but my understanding is
that Spring can cause a huge number of classes to be loaded
unnecessarily and it's quite easy to fill-up PermGen. Is there
anything you are using that inherently loads a large number of classes?
> I think it's a quite common scenario for a relatively small
> production environment but I don't find the best configuration that
> suits this type of deployment.
You can always choose to deploy applications in separate JVMs, etc. if
you find that one or more of them is a bit less stable than you'd
prefer. I find it's better to segregate such applications in order to
improve the stability of the other applications (that had been)
running in the same JVM.
Since you are using httpd, you have a lot of flexibility to re-route
requests to various back-ends which won't affect your clients.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTXlgdAAoJEBzwKT+lPKRYedoQAIhB84Et3LXr3B3lQFLpTMky
8OO4k6+9mlg5jeNCNrjMiKTT9VxfsRUZVQ9bI087XAVmvfWnvdomo/C/E14lpqeS
iNDqbL+BTKPGb13dh7ZMXe3OUbHHPGF/R93RHtjJnkw/7wgUbIXUWVj4Uhw5CdWf
p3Lhb3xcCF6XO/tIXRIGaTb9MNps6JSe2tJdOIjBUGR002GkFEFTOI5RPxYC9rb0
0UI3DPp53zF2jl3NSB8LsfkXBmyJCAbL+prpdWi3p2auH1OxmiTvkWP5muew2iR6
GFGSM3t/KwqV5bDdO0BC9SlPNr1NPtUJPtaTu8AsawPRFHnLdjJT8uPBcUdue0KJ
8D/yg446qtx4BVuGp2hs9Zob1yKu/It0/l6T7vKOZ8J6QWRmjVtEICWz6QQ2iRhs
1PzmUjK5VYc6792tX3se6L+4Ac7ASGuube3nuGOtf5+c4GALnLbLUJqV2c87GKrp
ZR3TugU3mqYkgcd4UxuIgsRbruy6QlI5y8w+3O45mcp3wSrCeIEjblgFbGQQPeBx
oQrseguWFUNEbMVwrcN+V+fc/O+c6dTJSw2mhDU5ix1bw2gBZ1CRdyJRiivMWUj3
VmUV3l8mddOx+zOn25ifZ76eSjvDLTVFcGw213QUg3YFF93leF/ePLde5U5V148h
OTfprjcAWf1LZs4/tCC5
=HKV0
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[OT] Re: Tomcat configuration with multiple webapps
Posted by Mark Eggers <it...@yahoo.com>.
On 5/4/2014 11:25 PM, Baldur wrote:
> Thank you Mark, good points. In fact, I had configured memory limits
> with JAVA_OPTS variable. I haven't used JMeter yet because
> applications need javascript, X509 authentication and so forth. So at
> first glance, it seemed somewhat difficult to test it. But I'll give
> it a try again.
>
> Thnx very much
Agrred, extensiver Javascript testing with JMeter is somewhat of a
challenge. You might look at the following steps:
1. Selenium
2. Export tests to JUnit
3. Run the JUnit tests stand-alone to see if they give you rational
results
4. Import the JUnit tests into JMeter
May be a lot more work than it's worth.
I don't know how to test X509 authentication. Someone else on the list
might.
. . . . just my 2 cents
/mde/
>
> -----Mensaje original----- De: Mark Eggers
> [mailto:its_toasted@yahoo.com] Enviado el: lunes, 28 de abril de 2014
> 23:06 Para: Tomcat Users List Asunto: Re: Tomcat configuration with
> multiple webapps
>
> Comments inline:
>
> On 4/28/2014 1:38 PM, Baldur Dae wrote:
>> Hi guys,
>>
>> First of all, thank so much for your quick responses. I'm really
>> grateful ;)
>>
>> The scenario I've described is staging/QA, which has a single
>> machine running Apache httpd and another box running 2 Tomcat
>> instances (it is expected that production environment will have at
>> least 2 boxes for http and other 2 for Tomcats).
>>
>> Regarding hardware both machines have 3GB RAM and they only run
>> some batch scripts at midnight. Apache httpd box is doing great so
>> far and I haven't noticed load problems yet.
>>
>> I should provide a 99% availability although this requirement might
>> be flexible to some extent. As far as I know, concurrent user rate
>> is expected to be low i.e. magnitude order = 100. One advantage
>> could be that users are located within the same timezone. Thus
>> there's a window for new deployments. Unfortunately, there will be
>> some occassions when a war will have to be redeployed transparently
>> to final users.
>>
>> All sessions are small since they store identity attributes.
>> Besides, all webapps are being migrated so that they use the same
>> authorization mechanism (Jasig CAS single sign on). Nevertheless,
>> JSF applications use ViewState which is ultimately stored in
>> session. Luckily, web services are stateless.
>>
>> With regard to versions, applications developed "in-house" use the
>> same versions (Spring 3, JSF 2, Hibernate 3) but this is not
>> always the standard stack due to some legacy apps.
>>
>> In order to face memory issues I've tweaked JAVA_OPTS to configure
>> memory limits. I tend to think that many problems come down to
>> memory leaks
>
> I would recommend that you set memory limits using CATALINA_OPTS
> (create a setenv.sh to set this). If you set JAVA_OPTS, then the
> shutdown task will also inherit your memory limits and could cause
> issues on memory-constrained systems.
>
>> produced by applications which must be fixed. Indeed Spring loads
>> a lot of classes but I guess this framework, as well as Hibernate
>> or JSF, are optimized enough, at least in this low-demanding
>> scenario.
>>
>> So, bearing in mind your advice and other ideas I was thinking of
>> my current roadmap would comprise: - Start out by a minimal set of
>> wars and add every war gradually to detect possible "big problems"
>> - Partition wars into 2 different groups: webapps and webservices -
>> Migrate the whole environment to Tomcat 7 - Evaluate different
>> connectors: BIO, NIO, APR - Find better tools to monitor/profile
>> applications to get a deeper insight about what's going on
>>
>> Thanks very much for your valuable information
>>
>> Cheers
>
> Sounds like a good plan.
>
> One thing you might consider is to set up some JMeter systems to
> replay load scenarios from production (if possible). Unless your
> integration tests are pretty good, you might not catch some memory /
> performance / threading issues until you get into production. Stress
> testing with a good sample load and JMeter is one way of ferreting
> out these issues.
>
>>
>>
>>
>>
>> 2014-04-28 16:09 GMT+02:00 Neven Cvetkovic
>> <ne...@gmail.com>:
>>
>>> Hey Baldur,
>>>
>>> On Mon, Apr 28, 2014 at 8:00 AM, Baldur <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'd like to get some help about my current architecture. The
>>>> current scenario uses mod_jk to connect Apache httpd and
>>>> Tomcat6. I have two Tomcat instances (using DeltaManager for
>>>> session replication and sticky session enabled) in order to
>>>> provide high availability and balance load across instances.
>>>> Currently Tomcat manages 28 webapps and 7 of them are only web
>>>> services. Generally speaking, a webapp usually involves JSF or
>>>> Struts while a web services war involves JAX-WS. Both types of
>>> application
>>>> have a common stack implemented with Spring and Hibernate. As
>>>> a result, each application produces a war of around 40-50 MB.
>>>>
>>>>
>>> Here are some questions that can make us better understand your
>>> environment and further the discussion on your choices:
>>>
>>> 1. What kind of hardware do you run these two instances on
>>> (single box, 2 boxes, how much RAM, etc..)? Do you have resources
>>> to run more Tomcat instances on this(these) box(es)?
>>>
>>> 2. Do you have HA as requirement for all the apps? Do you have
>>> any specific SLAs (service level agreements) you need to
>>> maintain?
>>>
>>> 3. Can you live without session replication, and just live with
>>> the sticky sessions? What kind of data do you keep in your
>>> sessions? How big are these sessions?
>>>
>>> 4. What's the order of magnitude for your concurrent users
>>> (100s, 1000s, 10000s) for these applications? I.e how many
>>> concurrent sessions do you need to maintain?
>>>
>>> 5. Are your webservices stateless, most of them usually are?
>>>
>>> 6. Do these applications share any libraries (Hibernate,
>>> Struts2, Spring, etc...)? What is the upgrade/release cycle for
>>> these applications? How do you deal with differences in
>>> versioning, e.g. Hibernate3 vs Hibernate4, or Spring 3.0 vs
>>> Spring 3.2 vs Spring 4.0, etc...
>>>
>>> In the ideal world, with infinite amount of resources (hardware,
>>> staff, etc) - I would have one Tomcat instance (or one cluster)
>>> per application, so I can segregate and isolate my application
>>> environments (JVMs). However, given huge number of applications,
>>> and that we don't have that much money to spare - that
>>> segregation might be too extreme, too wasteful - so we typically
>>> organize our applications to co-exist on the Tomcat instance(s),
>>> based on their importance, SLA agreements, release lifecycle,
>>> business operations, etc.
>>>
>>>
>>>
>>>> I'd like to ask you several questions to provide better
>>>> performance:
>>>>
>>>> * Which approach would be appropriate for this
>>>> scenario? All wars in one cluster? Maybe move web services to
>>>> other cluster?
>>>>
>>>>
>>> It might be useful to move webservices to a separate cluster
>>> that might not need session replication. You might gain some
>>> performance benefit by not having to replicate sessions across
>>> cluster members. Though, having 28 webapps (wars) on the same
>>> instance (clustered instance), my concern is isolation. What
>>> happens if one application trashes one of your JVMs? Then all
>>> other 27 apps are going to suffer and stress your other JVM. If
>>> you truly need HA, consider moving these apps on their own
>>> environment, independent of other apps.
>>>
>>>
>>>
>>>> * In order to improve deployments, which technique can
>>>> I use to minimize war size? Will be the cause of memory issues?
>>>> I have tried to
>>> put
>>>> some common jars (spring, apache-commons and so on) in Tomcat
>>>> lib but I don't know if there is a better approach by other
>>>> means.
>>>>
>>>>
>>> Have you observed any issues with the sizing of the apps, e.g.
>>> OutOfMemoryError (permgen space)? Ultimately, if you deploy ton
>>> of applications, and they all have ton of third-party libraries
>>> (think Spring, Hibernate, etc.) - you will end up with larger
>>> PermGen consumption, which might be exhausted after N
>>> applications.
>>>
>>> Placing shared libraries in the Tomcat shared folder might help
>>> with memory sizing issues, but then you face the upgrade
>>> lifecycle issue. You will need to coordinate the application
>>> upgrade properly. Also, you might end up with weird errors -
>>> because frameworks might share some objects statically, and
>>> that's not what your intent was, etc. Thus, using shared
>>> libraries need to be carefully planned. Usually, benefits of
>>> shared libraries are not worth the trouble, so we end up
>>> packaging each application separately. Shared libraries can be
>>> very useful when admins want to enforce library versioning, and
>>> force developers to use given environment, rather than them
>>> including what they want/need. It's an architectural decision,
>>> not so much performance optimization decision.
>>>
>>>
>>>
>>>> I read as much as I can but I'm stuck trying to find the best
>>>> tools to monitor the system and tackle memory issues (such as
>>>> the dreaded
>>> PermGen).
>>>> I think it's a quite common scenario for a relatively small
>>>> production environment but I don't find the best configuration
>>>> that suits this type
>>> of
>>>> deployment.
>>>>
>>>>
>>> Well, you probably want to profile your application(s) and see
>>> how they perform under various configuration options (memory
>>> sizing, connector sizing, etc). That gets much easier when you
>>> have all apps segmented to different environments. Your HTTPD
>>> setup helps a lot, as your clients don't care where HTTPD sends
>>> the traffic in the backend, to two instances or to twenty-eight
>>> instances. There might be minimal or insignificant performance
>>> overhead in maintaining 2 or 28 backend Tomcat instances
>>> connections. However, I would probably want to measure that too
>>> and see how it behaves under real-life like traffic.
>>>
>>>
>>>> Any help would be much appreciated.
>>>>
>>>> Thanks very much in advance
>>>>
>>>>
>>> Hope these questions give you something to think about and
>>> revisit and justify your choices.
>>>
>>> Cheers! Neven
>>>
>>
>
> . . . . just my two cents /mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat configuration with multiple webapps
Posted by Baldur <ba...@gmail.com>.
Thank you Mark, good points. In fact, I had configured memory limits with JAVA_OPTS variable. I haven't used JMeter yet because applications need javascript, X509 authentication and so forth. So at first glance, it seemed somewhat difficult to test it. But I'll give it a try again.
Thnx very much
-----Mensaje original-----
De: Mark Eggers [mailto:its_toasted@yahoo.com]
Enviado el: lunes, 28 de abril de 2014 23:06
Para: Tomcat Users List
Asunto: Re: Tomcat configuration with multiple webapps
Comments inline:
On 4/28/2014 1:38 PM, Baldur Dae wrote:
> Hi guys,
>
> First of all, thank so much for your quick responses. I'm really
> grateful ;)
>
> The scenario I've described is staging/QA, which has a single machine
> running Apache httpd and another box running 2 Tomcat instances (it is
> expected that production environment will have at least 2 boxes for
> http and other 2 for Tomcats).
>
> Regarding hardware both machines have 3GB RAM and they only run some
> batch scripts at midnight. Apache httpd box is doing great so far and
> I haven't noticed load problems yet.
>
> I should provide a 99% availability although this requirement might be
> flexible to some extent. As far as I know, concurrent user rate is
> expected to be low i.e. magnitude order = 100. One advantage could be
> that users are located within the same timezone. Thus there's a window
> for new deployments. Unfortunately, there will be some occassions when
> a war will have to be redeployed transparently to final users.
>
> All sessions are small since they store identity attributes. Besides,
> all webapps are being migrated so that they use the same authorization
> mechanism (Jasig CAS single sign on). Nevertheless, JSF applications
> use ViewState which is ultimately stored in session. Luckily, web
> services are stateless.
>
> With regard to versions, applications developed "in-house" use the
> same versions (Spring 3, JSF 2, Hibernate 3) but this is not always
> the standard stack due to some legacy apps.
>
> In order to face memory issues I've tweaked JAVA_OPTS to configure
> memory limits. I tend to think that many problems come down to memory
> leaks
I would recommend that you set memory limits using CATALINA_OPTS (create a setenv.sh to set this). If you set JAVA_OPTS, then the shutdown task will also inherit your memory limits and could cause issues on memory-constrained systems.
> produced by applications which must be fixed. Indeed Spring loads a
> lot of classes but I guess this framework, as well as Hibernate or
> JSF, are optimized enough, at least in this low-demanding scenario.
>
> So, bearing in mind your advice and other ideas I was thinking of my
> current roadmap would comprise:
> - Start out by a minimal set of wars and add every war gradually to
> detect possible "big problems"
> - Partition wars into 2 different groups: webapps and webservices
> - Migrate the whole environment to Tomcat 7
> - Evaluate different connectors: BIO, NIO, APR
> - Find better tools to monitor/profile applications to get a deeper
> insight about what's going on
>
> Thanks very much for your valuable information
>
> Cheers
Sounds like a good plan.
One thing you might consider is to set up some JMeter systems to replay load scenarios from production (if possible). Unless your integration tests are pretty good, you might not catch some memory / performance / threading issues until you get into production. Stress testing with a good sample load and JMeter is one way of ferreting out these issues.
>
>
>
>
> 2014-04-28 16:09 GMT+02:00 Neven Cvetkovic <ne...@gmail.com>:
>
>> Hey Baldur,
>>
>> On Mon, Apr 28, 2014 at 8:00 AM, Baldur <ba...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to get some help about my current architecture.
>>> The
>>> current scenario uses mod_jk to connect Apache httpd and Tomcat6. I
>>> have two Tomcat instances (using DeltaManager for session
>>> replication and sticky session enabled) in order to provide high
>>> availability and balance load across instances. Currently Tomcat
>>> manages 28 webapps and 7 of them are only web services. Generally
>>> speaking, a webapp usually involves JSF or Struts while a web
>>> services war involves JAX-WS. Both types of
>> application
>>> have a common stack implemented with Spring and Hibernate. As a
>>> result, each application produces a war of around 40-50 MB.
>>>
>>>
>> Here are some questions that can make us better understand your
>> environment and further the discussion on your choices:
>>
>> 1. What kind of hardware do you run these two instances on (single
>> box, 2 boxes, how much RAM, etc..)? Do you have resources to run more
>> Tomcat instances on this(these) box(es)?
>>
>> 2. Do you have HA as requirement for all the apps? Do you have any
>> specific SLAs (service level agreements) you need to maintain?
>>
>> 3. Can you live without session replication, and just live with the
>> sticky sessions? What kind of data do you keep in your sessions? How
>> big are these sessions?
>>
>> 4. What's the order of magnitude for your concurrent users (100s,
>> 1000s,
>> 10000s) for these applications? I.e how many concurrent sessions do
>> you need to maintain?
>>
>> 5. Are your webservices stateless, most of them usually are?
>>
>> 6. Do these applications share any libraries (Hibernate, Struts2,
>> Spring, etc...)? What is the upgrade/release cycle for these
>> applications? How do you deal with differences in versioning, e.g.
>> Hibernate3 vs Hibernate4, or Spring 3.0 vs Spring 3.2 vs Spring 4.0, etc...
>>
>> In the ideal world, with infinite amount of resources (hardware,
>> staff,
>> etc) - I would have one Tomcat instance (or one cluster) per
>> application, so I can segregate and isolate my application
>> environments (JVMs). However, given huge number of applications, and
>> that we don't have that much money to spare - that segregation might
>> be too extreme, too wasteful - so we typically organize our
>> applications to co-exist on the Tomcat instance(s), based on their
>> importance, SLA agreements, release lifecycle, business operations, etc.
>>
>>
>>
>>> I'd like to ask you several questions to provide
>>> better
>>> performance:
>>>
>>> * Which approach would be appropriate for this scenario? All wars
>>> in
>>> one cluster? Maybe move web services to other cluster?
>>>
>>>
>> It might be useful to move webservices to a separate cluster that
>> might not need session replication. You might gain some performance
>> benefit by not having to replicate sessions across cluster members.
>> Though, having 28 webapps (wars) on the same instance (clustered
>> instance), my concern is isolation. What happens if one application
>> trashes one of your JVMs? Then all other 27 apps are going to suffer
>> and stress your other JVM. If you truly need HA, consider moving
>> these apps on their own environment, independent of other apps.
>>
>>
>>
>>> * In order to improve deployments, which technique can I use to
>>> minimize war size? Will be the cause of memory issues? I have tried
>>> to
>> put
>>> some common jars (spring, apache-commons and so on) in Tomcat lib
>>> but I don't know if there is a better approach by other means.
>>>
>>>
>> Have you observed any issues with the sizing of the apps, e.g.
>> OutOfMemoryError (permgen space)? Ultimately, if you deploy ton of
>> applications, and they all have ton of third-party libraries (think
>> Spring, Hibernate, etc.) - you will end up with larger PermGen
>> consumption, which might be exhausted after N applications.
>>
>> Placing shared libraries in the Tomcat shared folder might help with
>> memory sizing issues, but then you face the upgrade lifecycle issue.
>> You will need to coordinate the application upgrade properly. Also,
>> you might end up with weird errors - because frameworks might share
>> some objects statically, and that's not what your intent was, etc.
>> Thus, using shared libraries need to be carefully planned. Usually,
>> benefits of shared libraries are not worth the trouble, so we end up
>> packaging each application separately. Shared libraries can be very
>> useful when admins want to enforce library versioning, and force
>> developers to use given environment, rather than them including what
>> they want/need. It's an architectural decision, not so much performance optimization decision.
>>
>>
>>
>>> I read as much as I can but I'm stuck trying to find the best tools
>>> to monitor the system and tackle memory issues (such as the dreaded
>> PermGen).
>>> I think it's a quite common scenario for a relatively small
>>> production environment but I don't find the best configuration that
>>> suits this type
>> of
>>> deployment.
>>>
>>>
>> Well, you probably want to profile your application(s) and see how
>> they perform under various configuration options (memory sizing,
>> connector sizing, etc). That gets much easier when you have all apps
>> segmented to different environments. Your HTTPD setup helps a lot, as
>> your clients don't care where HTTPD sends the traffic in the backend,
>> to two instances or to twenty-eight instances. There might be minimal
>> or insignificant performance overhead in maintaining 2 or 28 backend Tomcat instances connections.
>> However, I would probably want to measure that too and see how it
>> behaves under real-life like traffic.
>>
>>
>>> Any help would be much appreciated.
>>>
>>> Thanks very much in advance
>>>
>>>
>> Hope these questions give you something to think about and revisit
>> and justify your choices.
>>
>> Cheers!
>> Neven
>>
>
. . . . just my two cents
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat configuration with multiple webapps
Posted by Mark Eggers <it...@yahoo.com>.
Comments inline:
On 4/28/2014 1:38 PM, Baldur Dae wrote:
> Hi guys,
>
> First of all, thank so much for your quick responses. I'm really grateful ;)
>
> The scenario I've described is staging/QA, which has a single machine
> running Apache httpd and another box running 2 Tomcat instances (it is
> expected that production environment will have at least 2 boxes for http
> and other 2 for Tomcats).
>
> Regarding hardware both machines have 3GB RAM and they only run some batch
> scripts at midnight. Apache httpd box is doing great so far and I haven't
> noticed load problems yet.
>
> I should provide a 99% availability although this requirement might be
> flexible to some extent. As far as I know, concurrent user rate is expected
> to be low i.e. magnitude order = 100. One advantage could be that users are
> located within the same timezone. Thus there's a window for new
> deployments. Unfortunately, there will be some occassions when a war will
> have to be redeployed transparently to final users.
>
> All sessions are small since they store identity attributes. Besides, all
> webapps are being migrated so that they use the same authorization
> mechanism (Jasig CAS single sign on). Nevertheless, JSF applications use
> ViewState which is ultimately stored in session. Luckily, web services are
> stateless.
>
> With regard to versions, applications developed "in-house" use the same
> versions (Spring 3, JSF 2, Hibernate 3) but this is not always the standard
> stack due to some legacy apps.
>
> In order to face memory issues I've tweaked JAVA_OPTS to configure memory
> limits. I tend to think that many problems come down to memory leaks
I would recommend that you set memory limits using CATALINA_OPTS (create
a setenv.sh to set this). If you set JAVA_OPTS, then the shutdown task
will also inherit your memory limits and could cause issues on
memory-constrained systems.
> produced by applications which must be fixed. Indeed Spring loads a lot of
> classes but I guess this framework, as well as Hibernate or JSF, are
> optimized enough, at least in this low-demanding scenario.
>
> So, bearing in mind your advice and other ideas I was thinking of my
> current roadmap would comprise:
> - Start out by a minimal set of wars and add every war gradually to detect
> possible "big problems"
> - Partition wars into 2 different groups: webapps and webservices
> - Migrate the whole environment to Tomcat 7
> - Evaluate different connectors: BIO, NIO, APR
> - Find better tools to monitor/profile applications to get a deeper insight
> about what's going on
>
> Thanks very much for your valuable information
>
> Cheers
Sounds like a good plan.
One thing you might consider is to set up some JMeter systems to replay
load scenarios from production (if possible). Unless your integration
tests are pretty good, you might not catch some memory / performance /
threading issues until you get into production. Stress testing with a
good sample load and JMeter is one way of ferreting out these issues.
>
>
>
>
> 2014-04-28 16:09 GMT+02:00 Neven Cvetkovic <ne...@gmail.com>:
>
>> Hey Baldur,
>>
>> On Mon, Apr 28, 2014 at 8:00 AM, Baldur <ba...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to get some help about my current architecture.
>>> The
>>> current scenario uses mod_jk to connect Apache httpd and Tomcat6. I have
>>> two
>>> Tomcat instances (using DeltaManager for session replication and sticky
>>> session enabled) in order to provide high availability and balance load
>>> across instances. Currently Tomcat manages 28 webapps and 7 of them are
>>> only web services. Generally speaking, a webapp usually involves JSF or
>>> Struts while a web services war involves JAX-WS. Both types of
>> application
>>> have a common stack implemented with Spring and Hibernate. As a result,
>>> each
>>> application produces a war of around 40-50 MB.
>>>
>>>
>> Here are some questions that can make us better understand your environment
>> and further the discussion on your choices:
>>
>> 1. What kind of hardware do you run these two instances on (single box, 2
>> boxes, how much RAM, etc..)? Do you have resources to run more Tomcat
>> instances on this(these) box(es)?
>>
>> 2. Do you have HA as requirement for all the apps? Do you have any specific
>> SLAs (service level agreements) you need to maintain?
>>
>> 3. Can you live without session replication, and just live with the sticky
>> sessions? What kind of data do you keep in your sessions? How big are these
>> sessions?
>>
>> 4. What's the order of magnitude for your concurrent users (100s, 1000s,
>> 10000s) for these applications? I.e how many concurrent sessions do you
>> need to maintain?
>>
>> 5. Are your webservices stateless, most of them usually are?
>>
>> 6. Do these applications share any libraries (Hibernate, Struts2, Spring,
>> etc...)? What is the upgrade/release cycle for these applications? How do
>> you deal with differences in versioning, e.g. Hibernate3 vs Hibernate4, or
>> Spring 3.0 vs Spring 3.2 vs Spring 4.0, etc...
>>
>> In the ideal world, with infinite amount of resources (hardware, staff,
>> etc) - I would have one Tomcat instance (or one cluster) per application,
>> so I can segregate and isolate my application environments (JVMs). However,
>> given huge number of applications, and that we don't have that much money
>> to spare - that segregation might be too extreme, too wasteful - so we
>> typically organize our applications to co-exist on the Tomcat instance(s),
>> based on their importance, SLA agreements, release lifecycle, business
>> operations, etc.
>>
>>
>>
>>> I'd like to ask you several questions to provide better
>>> performance:
>>>
>>> * Which approach would be appropriate for this scenario? All wars
>>> in
>>> one cluster? Maybe move web services to other cluster?
>>>
>>>
>> It might be useful to move webservices to a separate cluster that might not
>> need session replication. You might gain some performance benefit by not
>> having to replicate sessions across cluster members. Though, having 28
>> webapps (wars) on the same instance (clustered instance), my concern is
>> isolation. What happens if one application trashes one of your JVMs? Then
>> all other 27 apps are going to suffer and stress your other JVM. If you
>> truly need HA, consider moving these apps on their own environment,
>> independent of other apps.
>>
>>
>>
>>> * In order to improve deployments, which technique can I use to
>>> minimize war size? Will be the cause of memory issues? I have tried to
>> put
>>> some common jars (spring, apache-commons and so on) in Tomcat lib but I
>>> don't know if there is a better approach by other means.
>>>
>>>
>> Have you observed any issues with the sizing of the apps, e.g.
>> OutOfMemoryError (permgen space)? Ultimately, if you deploy ton of
>> applications, and they all have ton of third-party libraries (think Spring,
>> Hibernate, etc.) - you will end up with larger PermGen consumption, which
>> might be exhausted after N applications.
>>
>> Placing shared libraries in the Tomcat shared folder might help with memory
>> sizing issues, but then you face the upgrade lifecycle issue. You will need
>> to coordinate the application upgrade properly. Also, you might end up with
>> weird errors - because frameworks might share some objects statically, and
>> that's not what your intent was, etc. Thus, using shared libraries need to
>> be carefully planned. Usually, benefits of shared libraries are not worth
>> the trouble, so we end up packaging each application separately. Shared
>> libraries can be very useful when admins want to enforce library
>> versioning, and force developers to use given environment, rather than them
>> including what they want/need. It's an architectural decision, not so much
>> performance optimization decision.
>>
>>
>>
>>> I read as much as I can but I'm stuck trying to find the best tools to
>>> monitor the system and tackle memory issues (such as the dreaded
>> PermGen).
>>> I think it's a quite common scenario for a relatively small production
>>> environment but I don't find the best configuration that suits this type
>> of
>>> deployment.
>>>
>>>
>> Well, you probably want to profile your application(s) and see how they
>> perform under various configuration options (memory sizing, connector
>> sizing, etc). That gets much easier when you have all apps segmented to
>> different environments. Your HTTPD setup helps a lot, as your clients don't
>> care where HTTPD sends the traffic in the backend, to two instances or to
>> twenty-eight instances. There might be minimal or insignificant performance
>> overhead in maintaining 2 or 28 backend Tomcat instances connections.
>> However, I would probably want to measure that too and see how it behaves
>> under real-life like traffic.
>>
>>
>>> Any help would be much appreciated.
>>>
>>> Thanks very much in advance
>>>
>>>
>> Hope these questions give you something to think about and revisit and
>> justify your choices.
>>
>> Cheers!
>> Neven
>>
>
. . . . just my two cents
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat configuration with multiple webapps
Posted by Baldur Dae <ba...@gmail.com>.
Hi guys,
First of all, thank so much for your quick responses. I'm really grateful ;)
The scenario I've described is staging/QA, which has a single machine
running Apache httpd and another box running 2 Tomcat instances (it is
expected that production environment will have at least 2 boxes for http
and other 2 for Tomcats).
Regarding hardware both machines have 3GB RAM and they only run some batch
scripts at midnight. Apache httpd box is doing great so far and I haven't
noticed load problems yet.
I should provide a 99% availability although this requirement might be
flexible to some extent. As far as I know, concurrent user rate is expected
to be low i.e. magnitude order = 100. One advantage could be that users are
located within the same timezone. Thus there's a window for new
deployments. Unfortunately, there will be some occassions when a war will
have to be redeployed transparently to final users.
All sessions are small since they store identity attributes. Besides, all
webapps are being migrated so that they use the same authorization
mechanism (Jasig CAS single sign on). Nevertheless, JSF applications use
ViewState which is ultimately stored in session. Luckily, web services are
stateless.
With regard to versions, applications developed "in-house" use the same
versions (Spring 3, JSF 2, Hibernate 3) but this is not always the standard
stack due to some legacy apps.
In order to face memory issues I've tweaked JAVA_OPTS to configure memory
limits. I tend to think that many problems come down to memory leaks
produced by applications which must be fixed. Indeed Spring loads a lot of
classes but I guess this framework, as well as Hibernate or JSF, are
optimized enough, at least in this low-demanding scenario.
So, bearing in mind your advice and other ideas I was thinking of my
current roadmap would comprise:
- Start out by a minimal set of wars and add every war gradually to detect
possible "big problems"
- Partition wars into 2 different groups: webapps and webservices
- Migrate the whole environment to Tomcat 7
- Evaluate different connectors: BIO, NIO, APR
- Find better tools to monitor/profile applications to get a deeper insight
about what's going on
Thanks very much for your valuable information
Cheers
2014-04-28 16:09 GMT+02:00 Neven Cvetkovic <ne...@gmail.com>:
> Hey Baldur,
>
> On Mon, Apr 28, 2014 at 8:00 AM, Baldur <ba...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to get some help about my current architecture.
> > The
> > current scenario uses mod_jk to connect Apache httpd and Tomcat6. I have
> > two
> > Tomcat instances (using DeltaManager for session replication and sticky
> > session enabled) in order to provide high availability and balance load
> > across instances. Currently Tomcat manages 28 webapps and 7 of them are
> > only web services. Generally speaking, a webapp usually involves JSF or
> > Struts while a web services war involves JAX-WS. Both types of
> application
> > have a common stack implemented with Spring and Hibernate. As a result,
> > each
> > application produces a war of around 40-50 MB.
> >
> >
> Here are some questions that can make us better understand your environment
> and further the discussion on your choices:
>
> 1. What kind of hardware do you run these two instances on (single box, 2
> boxes, how much RAM, etc..)? Do you have resources to run more Tomcat
> instances on this(these) box(es)?
>
> 2. Do you have HA as requirement for all the apps? Do you have any specific
> SLAs (service level agreements) you need to maintain?
>
> 3. Can you live without session replication, and just live with the sticky
> sessions? What kind of data do you keep in your sessions? How big are these
> sessions?
>
> 4. What's the order of magnitude for your concurrent users (100s, 1000s,
> 10000s) for these applications? I.e how many concurrent sessions do you
> need to maintain?
>
> 5. Are your webservices stateless, most of them usually are?
>
> 6. Do these applications share any libraries (Hibernate, Struts2, Spring,
> etc...)? What is the upgrade/release cycle for these applications? How do
> you deal with differences in versioning, e.g. Hibernate3 vs Hibernate4, or
> Spring 3.0 vs Spring 3.2 vs Spring 4.0, etc...
>
> In the ideal world, with infinite amount of resources (hardware, staff,
> etc) - I would have one Tomcat instance (or one cluster) per application,
> so I can segregate and isolate my application environments (JVMs). However,
> given huge number of applications, and that we don't have that much money
> to spare - that segregation might be too extreme, too wasteful - so we
> typically organize our applications to co-exist on the Tomcat instance(s),
> based on their importance, SLA agreements, release lifecycle, business
> operations, etc.
>
>
>
> > I'd like to ask you several questions to provide better
> > performance:
> >
> > * Which approach would be appropriate for this scenario? All wars
> > in
> > one cluster? Maybe move web services to other cluster?
> >
> >
> It might be useful to move webservices to a separate cluster that might not
> need session replication. You might gain some performance benefit by not
> having to replicate sessions across cluster members. Though, having 28
> webapps (wars) on the same instance (clustered instance), my concern is
> isolation. What happens if one application trashes one of your JVMs? Then
> all other 27 apps are going to suffer and stress your other JVM. If you
> truly need HA, consider moving these apps on their own environment,
> independent of other apps.
>
>
>
> > * In order to improve deployments, which technique can I use to
> > minimize war size? Will be the cause of memory issues? I have tried to
> put
> > some common jars (spring, apache-commons and so on) in Tomcat lib but I
> > don't know if there is a better approach by other means.
> >
> >
> Have you observed any issues with the sizing of the apps, e.g.
> OutOfMemoryError (permgen space)? Ultimately, if you deploy ton of
> applications, and they all have ton of third-party libraries (think Spring,
> Hibernate, etc.) - you will end up with larger PermGen consumption, which
> might be exhausted after N applications.
>
> Placing shared libraries in the Tomcat shared folder might help with memory
> sizing issues, but then you face the upgrade lifecycle issue. You will need
> to coordinate the application upgrade properly. Also, you might end up with
> weird errors - because frameworks might share some objects statically, and
> that's not what your intent was, etc. Thus, using shared libraries need to
> be carefully planned. Usually, benefits of shared libraries are not worth
> the trouble, so we end up packaging each application separately. Shared
> libraries can be very useful when admins want to enforce library
> versioning, and force developers to use given environment, rather than them
> including what they want/need. It's an architectural decision, not so much
> performance optimization decision.
>
>
>
> > I read as much as I can but I'm stuck trying to find the best tools to
> > monitor the system and tackle memory issues (such as the dreaded
> PermGen).
> > I think it's a quite common scenario for a relatively small production
> > environment but I don't find the best configuration that suits this type
> of
> > deployment.
> >
> >
> Well, you probably want to profile your application(s) and see how they
> perform under various configuration options (memory sizing, connector
> sizing, etc). That gets much easier when you have all apps segmented to
> different environments. Your HTTPD setup helps a lot, as your clients don't
> care where HTTPD sends the traffic in the backend, to two instances or to
> twenty-eight instances. There might be minimal or insignificant performance
> overhead in maintaining 2 or 28 backend Tomcat instances connections.
> However, I would probably want to measure that too and see how it behaves
> under real-life like traffic.
>
>
> > Any help would be much appreciated.
> >
> > Thanks very much in advance
> >
> >
> Hope these questions give you something to think about and revisit and
> justify your choices.
>
> Cheers!
> Neven
>
Re: Tomcat configuration with multiple webapps
Posted by Neven Cvetkovic <ne...@gmail.com>.
Hey Baldur,
On Mon, Apr 28, 2014 at 8:00 AM, Baldur <ba...@gmail.com> wrote:
> Hi all,
>
> I'd like to get some help about my current architecture.
> The
> current scenario uses mod_jk to connect Apache httpd and Tomcat6. I have
> two
> Tomcat instances (using DeltaManager for session replication and sticky
> session enabled) in order to provide high availability and balance load
> across instances. Currently Tomcat manages 28 webapps and 7 of them are
> only web services. Generally speaking, a webapp usually involves JSF or
> Struts while a web services war involves JAX-WS. Both types of application
> have a common stack implemented with Spring and Hibernate. As a result,
> each
> application produces a war of around 40-50 MB.
>
>
Here are some questions that can make us better understand your environment
and further the discussion on your choices:
1. What kind of hardware do you run these two instances on (single box, 2
boxes, how much RAM, etc..)? Do you have resources to run more Tomcat
instances on this(these) box(es)?
2. Do you have HA as requirement for all the apps? Do you have any specific
SLAs (service level agreements) you need to maintain?
3. Can you live without session replication, and just live with the sticky
sessions? What kind of data do you keep in your sessions? How big are these
sessions?
4. What's the order of magnitude for your concurrent users (100s, 1000s,
10000s) for these applications? I.e how many concurrent sessions do you
need to maintain?
5. Are your webservices stateless, most of them usually are?
6. Do these applications share any libraries (Hibernate, Struts2, Spring,
etc...)? What is the upgrade/release cycle for these applications? How do
you deal with differences in versioning, e.g. Hibernate3 vs Hibernate4, or
Spring 3.0 vs Spring 3.2 vs Spring 4.0, etc...
In the ideal world, with infinite amount of resources (hardware, staff,
etc) - I would have one Tomcat instance (or one cluster) per application,
so I can segregate and isolate my application environments (JVMs). However,
given huge number of applications, and that we don't have that much money
to spare - that segregation might be too extreme, too wasteful - so we
typically organize our applications to co-exist on the Tomcat instance(s),
based on their importance, SLA agreements, release lifecycle, business
operations, etc.
> I'd like to ask you several questions to provide better
> performance:
>
> * Which approach would be appropriate for this scenario? All wars
> in
> one cluster? Maybe move web services to other cluster?
>
>
It might be useful to move webservices to a separate cluster that might not
need session replication. You might gain some performance benefit by not
having to replicate sessions across cluster members. Though, having 28
webapps (wars) on the same instance (clustered instance), my concern is
isolation. What happens if one application trashes one of your JVMs? Then
all other 27 apps are going to suffer and stress your other JVM. If you
truly need HA, consider moving these apps on their own environment,
independent of other apps.
> * In order to improve deployments, which technique can I use to
> minimize war size? Will be the cause of memory issues? I have tried to put
> some common jars (spring, apache-commons and so on) in Tomcat lib but I
> don't know if there is a better approach by other means.
>
>
Have you observed any issues with the sizing of the apps, e.g.
OutOfMemoryError (permgen space)? Ultimately, if you deploy ton of
applications, and they all have ton of third-party libraries (think Spring,
Hibernate, etc.) - you will end up with larger PermGen consumption, which
might be exhausted after N applications.
Placing shared libraries in the Tomcat shared folder might help with memory
sizing issues, but then you face the upgrade lifecycle issue. You will need
to coordinate the application upgrade properly. Also, you might end up with
weird errors - because frameworks might share some objects statically, and
that's not what your intent was, etc. Thus, using shared libraries need to
be carefully planned. Usually, benefits of shared libraries are not worth
the trouble, so we end up packaging each application separately. Shared
libraries can be very useful when admins want to enforce library
versioning, and force developers to use given environment, rather than them
including what they want/need. It's an architectural decision, not so much
performance optimization decision.
> I read as much as I can but I'm stuck trying to find the best tools to
> monitor the system and tackle memory issues (such as the dreaded PermGen).
> I think it's a quite common scenario for a relatively small production
> environment but I don't find the best configuration that suits this type of
> deployment.
>
>
Well, you probably want to profile your application(s) and see how they
perform under various configuration options (memory sizing, connector
sizing, etc). That gets much easier when you have all apps segmented to
different environments. Your HTTPD setup helps a lot, as your clients don't
care where HTTPD sends the traffic in the backend, to two instances or to
twenty-eight instances. There might be minimal or insignificant performance
overhead in maintaining 2 or 28 backend Tomcat instances connections.
However, I would probably want to measure that too and see how it behaves
under real-life like traffic.
> Any help would be much appreciated.
>
> Thanks very much in advance
>
>
Hope these questions give you something to think about and revisit and
justify your choices.
Cheers!
Neven