You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by lightbulb432 <ve...@hotmail.com> on 2007/04/26 07:03:24 UTC
Seamless transition between application updates
What are the strategies you might use to update an application running on
multiple Tomcat instances (clustered with Apache mod_jk) with a seamless
transition for somebody who's using the application at the time the
application is redeployed? I've always wondered how web applications that
cannot afford downtime do it.
In my naive understanding I'd simply shut off all servers, redeploy the
updated applications, then turn all the servers back on. In this case, it's
pretty straightforward to understand what would happen. However, users would
experience downtime (or at least a non-seamless transition). In environments
that have requirements not to make users relogin everytime a change is made,
how would applications that provide smaller, more frequent updates survive?
How would you go about doing this, specifically, in a clustered Tomcat
environment? What things should I keep in mind when figuring out how to best
do this?
Problems that I foresee include someone who's POSTing from his current page
to a just-updated version of a servlet on Tomcat...in this case, what the
user expects to happen might not be what is expected, due to the servlet
update. In the best case the user has to relogin due to an error. In the
worst case the operation goes through successfully but modifies the data in
some unexpected way.
Your insight on this matter is greatly appreciated.
--
View this message in context: http://www.nabble.com/Seamless-transition-between-application-updates-tf3649567.html#a10194118
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Seamless transition between application updates
Posted by Tim Lucia <ti...@yahoo.com>.
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: Thursday, April 26, 2007 8:54 AM
> To: Tomcat Users List
> Subject: Re: Seamless transition between application updates
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Lightbulb,
>
> lightbulb432 wrote:
> > What are the strategies you might use to update an application
> > running on multiple Tomcat instances (clustered with Apache mod_jk)
> > with a seamless transition for somebody who's using the application
> > at the time the application is redeployed? I've always wondered how
> > web applications that cannot afford downtime do it.
>
> The biggest problem isn't the code, it's the other stuff that needs to
> be upgraded. Tim describes a good way to upgrade app servers
> incrementally, but doesn't say anything about a database.
>
> Usually, a major upgrade involves changes to a database which are
> incompatible with the old version. Crafting a deployment plan to upgrade
> the database and the application in phases is challenging.
In our case, we make minor, incremental changes all the time. Major changes
do require a 'maintenance' interval. In a previous company, they had
contractual arrangements to do exactly that -- Sunday afternoon for 2 hours
and Wednesday at 1AM for one hour. In this case upgrades often included
schema changes.
My current company provide services for school districts and we have a
/very/ predictable utilization pattern so there are times during the day and
on the weekend where we have only one or two users. The database does tend
to constrain the way in which application changes are made. This may be
good, or bad.
>
> The process of upgrading Tomcat instances in a cluster is much easier.
> Note that anyone who /was/ using the old version and suddenly finds
> themselves using the new version of the application might get
> ClassCastExceptions or something similar if your session objects do not
> have fixed serialVersionIDs.
>
> > Problems that I foresee include someone who's POSTing from his
> > current page to a just-updated version of a servlet on Tomcat...in
> > this case, what the user expects to happen might not be what is
> > expected, due to the servlet update. In the best case the user has to
> > relogin due to an error. In the worst case the operation goes through
> > successfully but modifies the data in some unexpected way.
>
> You probably want to herd users over to a set of servers that will
> continue to run the old version for a while. Basically, disable logins
> on that server and direct all new mod_jk traffic to the "old" app
> servers (not sure how to do this... you'll have to ask the mod_jk gurus,
> or consult your load balancer documentation).
There are a few relevant things here:
- You can version your workers
- You can stop your workers (ALL traffic), can be done using /jkmanager aka
/jkstatus
- You can disable your workers (all NEW traffic)
Pretty much everything I learned about mod_jk came from the online
documentation and experimentation. People always cry that it's hard to use,
but I didn't find it that difficult. Getting it to work correctly on IIS 6
with application pools was a little bit tricky, but I'm (thankfully!) on a
100% Linux stack now -- Red Hat, Apache, Tomcat, Java, MySQL. When I
started this job 18 months ago, it was Windows, JRun 3, IIS, and Oracle.
Having said all this, *I* would enjoy reading how others do their upgrades.
Tim
> After the users die down on the servers marked for the first upgrade, go
> ahead and upgrade their code. Then, re-enable new traffic to the
> upgraded server. All of the old sessions should still point to the old
> server (if you are using session affinity, which will really help during
> this process). Now, disable logins on the old servers and eventually
> they will go idle. At that point, you can upgrade the remaining servers.
>
> You can basically do this in as many phases as you want, so if you have
> 15 machines, maybe you want to upgrade 5 at a time.
>
> Again, my biggest worry would be if there are any significant
> incompatibilities between code versions and their dependencies (such as
> database structure, etc.).
>
> Hope that helps,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.7 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQFGMKDR9CaO5/Lv0PARAkMBAJ9OqKjQn/K3TOxWaqaaD6qrNaBuPQCgqmOe
> WtTEmt56VxCMyzZ5g79FEBo=
> =G0w+
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Seamless transition between application updates
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Lightbulb,
lightbulb432 wrote:
> What are the strategies you might use to update an application
> running on multiple Tomcat instances (clustered with Apache mod_jk)
> with a seamless transition for somebody who's using the application
> at the time the application is redeployed? I've always wondered how
> web applications that cannot afford downtime do it.
The biggest problem isn't the code, it's the other stuff that needs to
be upgraded. Tim describes a good way to upgrade app servers
incrementally, but doesn't say anything about a database.
Usually, a major upgrade involves changes to a database which are
incompatible with the old version. Crafting a deployment plan to upgrade
the database and the application in phases is challenging.
The process of upgrading Tomcat instances in a cluster is much easier.
Note that anyone who /was/ using the old version and suddenly finds
themselves using the new version of the application might get
ClassCastExceptions or something similar if your session objects do not
have fixed serialVersionIDs.
> Problems that I foresee include someone who's POSTing from his
> current page to a just-updated version of a servlet on Tomcat...in
> this case, what the user expects to happen might not be what is
> expected, due to the servlet update. In the best case the user has to
> relogin due to an error. In the worst case the operation goes through
> successfully but modifies the data in some unexpected way.
You probably want to herd users over to a set of servers that will
continue to run the old version for a while. Basically, disable logins
on that server and direct all new mod_jk traffic to the "old" app
servers (not sure how to do this... you'll have to ask the mod_jk gurus,
or consult your load balancer documentation).
After the users die down on the servers marked for the first upgrade, go
ahead and upgrade their code. Then, re-enable new traffic to the
upgraded server. All of the old sessions should still point to the old
server (if you are using session affinity, which will really help during
this process). Now, disable logins on the old servers and eventually
they will go idle. At that point, you can upgrade the remaining servers.
You can basically do this in as many phases as you want, so if you have
15 machines, maybe you want to upgrade 5 at a time.
Again, my biggest worry would be if there are any significant
incompatibilities between code versions and their dependencies (such as
database structure, etc.).
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFGMKDR9CaO5/Lv0PARAkMBAJ9OqKjQn/K3TOxWaqaaD6qrNaBuPQCgqmOe
WtTEmt56VxCMyzZ5g79FEBo=
=G0w+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Seamless transition between application updates
Posted by lightbulb432 <ve...@hotmail.com>.
You describe a good solution. Is there a way to have multiple versions of an
application on one server, as opposed to having to divide up versions of the
application per server?
I ask because if your users tend to have long sessions, then in the Tomcat
cluster you'd like to disable you could be waiting hours for all those
sessions to die, in which case you're "wasting hardware" and overburdening
the version of the server with your new application.
I'm hoping there's a way to do this...
Thanks.
Tim Lucia wrote:
>
> At my company, we have a weekly application update. Sometimes it is more
> frequent than that. We have Apache/httpd + mod_jk and three clustered
> Tomcats. Updating the Tomcats one at a time causes no loss of service as
> the other two pick up. Of course we do this during the periods of lowest
> usage, and you cannot get away with incompatible session attribute
> changes.
> I have an "ant deploy" rule which iterates over a list of Tomcat hosts and
> undeploys-the-old/deploys-the-new using the Tomcat manager / deploy tasks.
>
> One alternative is to use mod_jk (or load balancer / traffic router of
> your
> choice) to partition the Tomcats in to 2 clusters: TC1 and TC2
>
> TC1 has V2.0.5 of /yourapp
> TC2 has V2.0.6 of /yourapp
>
> Enable/start the workers on TC2
> Disable each worker on TC1 and wait for the count to go to zero
>
> Upgrade TC1 to V2.0.7 of /yourapp
> Move traffic from TC2 to TC1
>
> I've tested the latter and it worked during testing, but we've not had
> sufficient need to make use of it in production.
>
> HTH,
> Tim
>
>
>> -----Original Message-----
>> From: lightbulb432 [mailto:veerukrishnan@hotmail.com]
>> Sent: Thursday, April 26, 2007 1:03 AM
>> To: users@tomcat.apache.org
>> Subject: Seamless transition between application updates
>>
>>
>> What are the strategies you might use to update an application running on
>> multiple Tomcat instances (clustered with Apache mod_jk) with a seamless
>> transition for somebody who's using the application at the time the
>> application is redeployed? I've always wondered how web applications that
>> cannot afford downtime do it.
>>
>> In my naive understanding I'd simply shut off all servers, redeploy the
>> updated applications, then turn all the servers back on. In this case,
>> it's
>> pretty straightforward to understand what would happen. However, users
>> would
>> experience downtime (or at least a non-seamless transition). In
>> environments
>> that have requirements not to make users relogin everytime a change is
>> made,
>> how would applications that provide smaller, more frequent updates
>> survive?
>>
>> How would you go about doing this, specifically, in a clustered Tomcat
>> environment? What things should I keep in mind when figuring out how to
>> best
>> do this?
>>
>> Problems that I foresee include someone who's POSTing from his current
>> page
>> to a just-updated version of a servlet on Tomcat...in this case, what the
>> user expects to happen might not be what is expected, due to the servlet
>> update. In the best case the user has to relogin due to an error. In the
>> worst case the operation goes through successfully but modifies the data
>> in
>> some unexpected way.
>>
>> Your insight on this matter is greatly appreciated.
>> --
>> View this message in context: http://www.nabble.com/Seamless-transition-
>> between-application-updates-tf3649567.html#a10194118
>> Sent from the Tomcat - User mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To start a new topic, e-mail: users@tomcat.apache.org
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/Seamless-transition-between-application-updates-tf3649567.html#a10434931
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Seamless transition between application updates
Posted by Tim Lucia <ti...@yahoo.com>.
At my company, we have a weekly application update. Sometimes it is more
frequent than that. We have Apache/httpd + mod_jk and three clustered
Tomcats. Updating the Tomcats one at a time causes no loss of service as
the other two pick up. Of course we do this during the periods of lowest
usage, and you cannot get away with incompatible session attribute changes.
I have an "ant deploy" rule which iterates over a list of Tomcat hosts and
undeploys-the-old/deploys-the-new using the Tomcat manager / deploy tasks.
One alternative is to use mod_jk (or load balancer / traffic router of your
choice) to partition the Tomcats in to 2 clusters: TC1 and TC2
TC1 has V2.0.5 of /yourapp
TC2 has V2.0.6 of /yourapp
Enable/start the workers on TC2
Disable each worker on TC1 and wait for the count to go to zero
Upgrade TC1 to V2.0.7 of /yourapp
Move traffic from TC2 to TC1
I've tested the latter and it worked during testing, but we've not had
sufficient need to make use of it in production.
HTH,
Tim
> -----Original Message-----
> From: lightbulb432 [mailto:veerukrishnan@hotmail.com]
> Sent: Thursday, April 26, 2007 1:03 AM
> To: users@tomcat.apache.org
> Subject: Seamless transition between application updates
>
>
> What are the strategies you might use to update an application running on
> multiple Tomcat instances (clustered with Apache mod_jk) with a seamless
> transition for somebody who's using the application at the time the
> application is redeployed? I've always wondered how web applications that
> cannot afford downtime do it.
>
> In my naive understanding I'd simply shut off all servers, redeploy the
> updated applications, then turn all the servers back on. In this case,
> it's
> pretty straightforward to understand what would happen. However, users
> would
> experience downtime (or at least a non-seamless transition). In
> environments
> that have requirements not to make users relogin everytime a change is
> made,
> how would applications that provide smaller, more frequent updates
> survive?
>
> How would you go about doing this, specifically, in a clustered Tomcat
> environment? What things should I keep in mind when figuring out how to
> best
> do this?
>
> Problems that I foresee include someone who's POSTing from his current
> page
> to a just-updated version of a servlet on Tomcat...in this case, what the
> user expects to happen might not be what is expected, due to the servlet
> update. In the best case the user has to relogin due to an error. In the
> worst case the operation goes through successfully but modifies the data
> in
> some unexpected way.
>
> Your insight on this matter is greatly appreciated.
> --
> View this message in context: http://www.nabble.com/Seamless-transition-
> between-application-updates-tf3649567.html#a10194118
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org