You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Bill Davidson <bi...@gmail.com> on 2008/09/17 23:05:33 UTC
Server Maintenance Across Timezones (global)
My company's main webapp is used around the world (Europe, North America,
Australia, etc.).
We're using Tomcat as our app server and Oracle (10g) for our database.
When we want to do an upgrade, that usually involves DDL changes to the
database as well as corresponding changes to the webapp which means we
have to make our users log out so we can shut down the app, update the
DDL and restart the updated webapp. The changes are interdependent.
It's all or nothing.
This was not a big problem when we were just doing business in the U.S.
We'd do upgrades late at night when nobody (or hardly anyone) was using
the system. The problem now is that late at night here is middle of
the day in other places and downtime in the middle of the day is a real
problem. Our customers use our app to run parts of their business so
downtime in the middle of the day is very very bad. They understandably
don't like telling their customers: "I'd like to help you but I need to
wait for the Americans to upgrade their systems."
I'm not sure how to deal with this. I've been trying to think of a way
to use multiple servers and multiple databases but that seems like a
synchronization nightmare. Losing data consistency is not an option.
I'm sure that plenty of others on this list have had to deal with this
problem. Any suggestions? How have others dealt with it?
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Server Maintenance Across Timezones (global)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Peng,
Peng Tuck Kwok wrote:
> There's a lot of good suggestions here, maybe you could also justify
> maintaining a separate instance for the American customers. That would at
> least allow at a minimum to roll out changes specific for them, conform to
> their maintenance time :P. Yes I do realize it would be a replication of
> code in terms of releases but it is something to think about.
For an application used this widely, presumably you'll need multiple
physical machines, anyway. Since you require multiple machines, this
"replication of code" will have to happen, anyway. You ought to be able
to configure certain servers/clusters to provide, say, US-oriented
content, while others provide content for other geographic areas.
I think that's quite an elegant solution, actually. Thanks, Paul!
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkjXmM8ACgkQ9CaO5/Lv0PCwBgCcCwv6GiGR/8HFRULnIwPDNoM2
uqwAoJMB363isCzb9VJ3rAjK2T35dfQa
=N+Tg
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Server Maintenance Across Timezones (global)
Posted by Peng Tuck Kwok <pe...@gmail.com>.
There's a lot of good suggestions here, maybe you could also justify
maintaining a separate instance for the American customers. That would at
least allow at a minimum to roll out changes specific for them, conform to
their maintenance time :P. Yes I do realize it would be a replication of
code in terms of releases but it is something to think about.
On Thu, Sep 18, 2008 at 5:05 AM, Bill Davidson <bi...@gmail.com> wrote:
> My company's main webapp is used around the world (Europe, North America,
> Australia, etc.).
>
> We're using Tomcat as our app server and Oracle (10g) for our database.
>
> When we want to do an upgrade, that usually involves DDL changes to the
> database as well as corresponding changes to the webapp which means we
> have to make our users log out so we can shut down the app, update the
> DDL and restart the updated webapp. The changes are interdependent.
> It's all or nothing.
>
> This was not a big problem when we were just doing business in the U.S.
> We'd do upgrades late at night when nobody (or hardly anyone) was using
> the system. The problem now is that late at night here is middle of
> the day in other places and downtime in the middle of the day is a real
> problem. Our customers use our app to run parts of their business so
> downtime in the middle of the day is very very bad. They understandably
> don't like telling their customers: "I'd like to help you but I need to
> wait for the Americans to upgrade their systems."
>
> I'm not sure how to deal with this. I've been trying to think of a way
> to use multiple servers and multiple databases but that seems like a
> synchronization nightmare. Losing data consistency is not an option.
>
> I'm sure that plenty of others on this list have had to deal with this
> problem. Any suggestions? How have others dealt with it?
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Server Maintenance Across Timezones (global)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Alan,
Alan Chaney wrote:
> As far as the database side of it goes it seems to me that much of it is
> a question of making the 'live-update' a design requirement for any
> upgrades. You have to make it possible for the changes to the database
> to 'co-exist' and then update the live database independently of the
> application. When the new tables are ready, deploy your new application
> and (hopefully) go home smiling.
This is the "right" way to do it, but it's a complete PITA to implement. :(
> I'd be interested to hear from people who have clustered solutions. Once
> again I suspect there may be problems with trying to sustain sessions
> across the upgrades.
We don't do this (operating only in the US for now), but one way to do
this is to have multiple clusters, and multiple clustered databases
using replication across clusters.
You remove one of the db clusters from the replication schedule and one
of the app server clusters from normal service, and let your clients
move to other machines. Then, you upgrade the segregated cluster while
your users use the machines and databases running your old stuff. Bring
this new cluster online and let users start using it.
Repeat with the other clusters, and then resume db replication. It's
tricky, but possible, especially if your users will be okay if some of
them run the old version while others get the new version -- /and/ if
they don't need to share data immediately (i.e. the time delay before
all databases are sync'd up is tolerable).
Then again, there's always the weekend, when most folks aren't working,
anyway ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkjSfrUACgkQ9CaO5/Lv0PA90gCfY6mqVVdb80FOjoyz95Dlb5Lt
+2gAni0v8tzPtG5IoDSAC37D0V4FiPx+
=lTF9
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Server Maintenance Across Timezones (global)
Posted by Alan Chaney <al...@compulsivecreative.com>.
Hi Bill
Yes, we have the same problem and I've been working on ways to improve
the situation. Unfortunately, I don't think there is an easy or simple
solution - a lot will depend upon your application.
As far as the database side of it goes it seems to me that much of it is
a question of making the 'live-update' a design requirement for any
upgrades. You have to make it possible for the changes to the database
to 'co-exist' and then update the live database independently of the
application. When the new tables are ready, deploy your new application
and (hopefully) go home smiling.
We tend to create DDL changes as SQL scripts - test them out on the
development systems and then run them on the live site. Some examples
make it easier to see what I mean.
A 'simple' change might be just adding an extra field with a static
default. Obviously easy - you just run the script, it adds the extra
field, the old app. doesn't have any knowledge of it. You make sure the
script correctly initializes the field. When the update app runs its there.
A more complex change is where a new field is required that may need to
be derived from existing data. Since this data may be changing on the
fly, you have to ensure consistency. The best way I've found for this is
to create an 'update' transaction in your ORM code (or whatever you are
using) that detects that the DDL state has changed and then runs an ACID
initialization of the new field.
Once again, the testing of this update is a key part of your release
testing strategy.
As for tomcat itself its rather going to depend whether you run
clustered or not.
One method that I've used on unclustered systems is to configure the new
system on a server instance on a different IP address on a system
fronted by httpd, set up some redirects in httpd and just, as it were,
'hup' from the old instance to the new instance.
This requires a DNS change, but any 'old' requests to the old server are
redirected to the new server. After about 48 hours you should be able to
shut down the old server.
However, we were able to catch an 'idle' period - you may not be able to
do that and you'd have to ensure that any sessions active on the old
server were correctly propagated to the new server.
I'd be interested to hear from people who have clustered solutions. Once
again I suspect there may be problems with trying to sustain sessions
across the upgrades.
Regards
Alan Chaney
Bill Davidson wrote:
> My company's main webapp is used around the world (Europe, North America,
> Australia, etc.).
>
> We're using Tomcat as our app server and Oracle (10g) for our database.
>
> When we want to do an upgrade, that usually involves DDL changes to the
> database as well as corresponding changes to the webapp which means we
> have to make our users log out so we can shut down the app, update the
> DDL and restart the updated webapp. The changes are interdependent.
> It's all or nothing.
>
> This was not a big problem when we were just doing business in the U.S.
> We'd do upgrades late at night when nobody (or hardly anyone) was using
> the system. The problem now is that late at night here is middle of
> the day in other places and downtime in the middle of the day is a real
> problem. Our customers use our app to run parts of their business so
> downtime in the middle of the day is very very bad. They understandably
> don't like telling their customers: "I'd like to help you but I need to
> wait for the Americans to upgrade their systems."
>
> I'm not sure how to deal with this. I've been trying to think of a way
> to use multiple servers and multiple databases but that seems like a
> synchronization nightmare. Losing data consistency is not an option.
>
> I'm sure that plenty of others on this list have had to deal with this
> problem. Any suggestions? How have others dealt with it?
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> !DSPAM:48d172ca19921381456296!
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Server Maintenance Across Timezones (global)
Posted by Martin Gainty <mg...@hotmail.com>.
John-
your approach works best for long term operational and maintenance considerations
layersa properly architected and designed enterprise system has ability to interchange either the UI or DB with minimal impact to the other layers except for predefined 'interface points' which AOP folks
call 'contract'
Unfortunately the majority of 'J2EE enterprise systems' which start as prototypes have evolved into monolithic un-architected and un-documented tangled mess..where everyone is afraid
to touch an attribute in one component of one layer which would crash the other layers
does anyone remember Structured Exception Handling?
Thanks,
Martin
______________________________________________
Disclaimer and confidentiality note
Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission.
> Date: Wed, 17 Sep 2008 16:08:19 -0700
> From: billdsd@gmail.com
> To: users@tomcat.apache.org
> Subject: Re: Server Maintenance Across Timezones (global)
>
> John5342 wrote:
> > I get around my the same kinds of problems by keeping all the layers of the
> > web app seperate so that i can swap them out one at a time and create a near
> > seemless upgrade. The layers in my web apps are:
> >
> > 1 The web interface.
> > 2. The application logic. (this may itself be several layers in itself if
> > the app is complicated)
> > 3. The database access layer.
> > 4. The database.
> >
> > [...]
> >
> > Hope what i said is useful.
> >
>
> I think it will be useful when we get to the point of redesigning the app
> from scratch. It's a bit tough to replace the data access layer of a large
> complex app that's been around for a long time though.
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
_________________________________________________________________
Want to do more with Windows Live? Learn “10 hidden secrets” from Jamie.
http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008
Re: Server Maintenance Across Timezones (global)
Posted by John5342 <jo...@googlemail.com>.
>I think it will be useful when we get to the point of redesigning the app
>from scratch. It's a bit tough to replace the data access layer of a large
>complex app that's been around for a long time though.
It is indeed difficult to change a long standing app but in the long run its
a better approach. Definately something for the next major overhaul. I use
the same design principals on all of my web apps. Even on my own custom
written servers (non http). I am currently in my 7th year of continuous
uptime with on average 10 structural app changes every month and i have to
do that on 64 servers. Trust me when i say the technique works and is well
worth investing the time to setup. Building the initial framework for it is
a pain but once its up and running then its well worth it and you will never
look back.
2008/9/18 Bill Davidson <bi...@gmail.com>
> John5342 wrote:
>
>> I get around my the same kinds of problems by keeping all the layers of
>> the
>> web app seperate so that i can swap them out one at a time and create a
>> near
>> seemless upgrade. The layers in my web apps are:
>>
>> 1 The web interface.
>> 2. The application logic. (this may itself be several layers in itself if
>> the app is complicated)
>> 3. The database access layer.
>> 4. The database.
>>
>> [...]
>>
>> Hope what i said is useful.
>>
>>
>
> I think it will be useful when we get to the point of redesigning the app
> from scratch. It's a bit tough to replace the data access layer of a large
> complex app that's been around for a long time though.
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Server Maintenance Across Timezones (global)
Posted by Bill Davidson <bi...@gmail.com>.
John5342 wrote:
> I get around my the same kinds of problems by keeping all the layers of the
> web app seperate so that i can swap them out one at a time and create a near
> seemless upgrade. The layers in my web apps are:
>
> 1 The web interface.
> 2. The application logic. (this may itself be several layers in itself if
> the app is complicated)
> 3. The database access layer.
> 4. The database.
>
> [...]
>
> Hope what i said is useful.
>
I think it will be useful when we get to the point of redesigning the app
from scratch. It's a bit tough to replace the data access layer of a large
complex app that's been around for a long time though.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Server Maintenance Across Timezones (global)
Posted by John5342 <jo...@googlemail.com>.
I get around my the same kinds of problems by keeping all the layers of the
web app seperate so that i can swap them out one at a time and create a near
seemless upgrade. The layers in my web apps are:
1 The web interface.
2. The application logic. (this may itself be several layers in itself if
the app is complicated)
3. The database access layer.
4. The database.
The key in your case is in the database access layer. The database access
layer should be programmed to read either database format and write to the
new one (filling in defaults/placeholders where necessary).
All you need to do then is setup the structure for the new database on a new
server (can use the server for testing while your at it). Drop in the new
database backend. Do the rest of the data migration in the background safe
in the knowledge that your webapp is still able to do whatever it needs with
the old data and is already sending data to your new database aswell. The
frontend can then be changed whenever (if at all).
The key to any seemless upgrade is layers in the same way that redundency
provides layers for server downtime.
Hope what i said is useful.
John5342
2008/9/17 Bill Davidson <bi...@gmail.com>
> My company's main webapp is used around the world (Europe, North America,
> Australia, etc.).
>
> We're using Tomcat as our app server and Oracle (10g) for our database.
>
> When we want to do an upgrade, that usually involves DDL changes to the
> database as well as corresponding changes to the webapp which means we
> have to make our users log out so we can shut down the app, update the
> DDL and restart the updated webapp. The changes are interdependent.
> It's all or nothing.
>
> This was not a big problem when we were just doing business in the U.S.
> We'd do upgrades late at night when nobody (or hardly anyone) was using
> the system. The problem now is that late at night here is middle of
> the day in other places and downtime in the middle of the day is a real
> problem. Our customers use our app to run parts of their business so
> downtime in the middle of the day is very very bad. They understandably
> don't like telling their customers: "I'd like to help you but I need to
> wait for the Americans to upgrade their systems."
>
> I'm not sure how to deal with this. I've been trying to think of a way
> to use multiple servers and multiple databases but that seems like a
> synchronization nightmare. Losing data consistency is not an option.
>
> I'm sure that plenty of others on this list have had to deal with this
> problem. Any suggestions? How have others dealt with it?
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: Server Maintenance Across Timezones (global)
Posted by Paul McGurn <Pa...@logmein.com>.
Easy solution to that one:
Make order ID's GUIDs (globally unique identifiers). Most platforms allow for this, so I presume that Java is one of them. This could also be achieved by adding another element to the order ID generation as well. For instance, if your order ID was comprised of a few letters and the data, you could multiply the numeric part by a random number, or something to that effect.
The only hole in that though is pre-existing data, but realistically, if you don't already have duplicates, it should not be a problem. The problem would arise if you have any dependency in the application on how the ID if formatted. If that's your only concern, you're doing good so far!
Paul McGurn
-----Original Message-----
From: Bill Davidson [mailto:billdsd@gmail.com]
Sent: Wednesday, September 17, 2008 6:45 PM
To: Tomcat Users List
Subject: Re: Server Maintenance Across Timezones (global)
Paul McGurn wrote:
> Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific "cluster" of their cloud, and then everything that customer does in the application is tied back to that "cluster". The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data.
>
> By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is.
>
Indeed.
> In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds.
>
The main difficulty with this is consistency. Many parts of our data are
tagged with dynamically generated id's (order id's for example) that
are printed out and referenced by our customers, and even their customers.
Running on multiple databases allows for the possibility (at this
point certainty) of generating duplicate ids across different regions.
This could result in a lot of confusion, particularly for support calls.
We may need to learn to live with that but I still am not crazy about it.
This may be the only reasonable way to do this without completely
re-architecting our app (which is not really doable any time soon).
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Server Maintenance Across Timezones (global)
Posted by Bill Davidson <bi...@gmail.com>.
Paul McGurn wrote:
> Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific "cluster" of their cloud, and then everything that customer does in the application is tied back to that "cluster". The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data.
>
> By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is.
>
Indeed.
> In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds.
>
The main difficulty with this is consistency. Many parts of our data are
tagged with dynamically generated id's (order id's for example) that
are printed out and referenced by our customers, and even their customers.
Running on multiple databases allows for the possibility (at this
point certainty) of generating duplicate ids across different regions.
This could result in a lot of confusion, particularly for support calls.
We may need to learn to live with that but I still am not crazy about it.
This may be the only reasonable way to do this without completely
re-architecting our app (which is not really doable any time soon).
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Server Maintenance Across Timezones (global)
Posted by Paul McGurn <Pa...@logmein.com>.
I think the ideal approach here (potentially) is segregating your customer base. Here's an idea directly from how Salesforce.com does it.
Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific "cluster" of their cloud, and then everything that customer does in the application is tied back to that "cluster". The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data.
By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is.
In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds.
The only alternative I can personally offer is ensuring that the intended webapp upgrade is backwards compatible, and that the intended database upgrade is backwards/forwards compatible, so you can roll them separately (which would probably be more of a challenge than geo-separate environments).
Paul McGurn
-----Original Message-----
From: Bill Davidson [mailto:billdsd@gmail.com]
Sent: Wednesday, September 17, 2008 5:06 PM
To: Tomcat Users List
Subject: Server Maintenance Across Timezones (global)
My company's main webapp is used around the world (Europe, North America,
Australia, etc.).
We're using Tomcat as our app server and Oracle (10g) for our database.
When we want to do an upgrade, that usually involves DDL changes to the
database as well as corresponding changes to the webapp which means we
have to make our users log out so we can shut down the app, update the
DDL and restart the updated webapp. The changes are interdependent.
It's all or nothing.
This was not a big problem when we were just doing business in the U.S.
We'd do upgrades late at night when nobody (or hardly anyone) was using
the system. The problem now is that late at night here is middle of
the day in other places and downtime in the middle of the day is a real
problem. Our customers use our app to run parts of their business so
downtime in the middle of the day is very very bad. They understandably
don't like telling their customers: "I'd like to help you but I need to
wait for the Americans to upgrade their systems."
I'm not sure how to deal with this. I've been trying to think of a way
to use multiple servers and multiple databases but that seems like a
synchronization nightmare. Losing data consistency is not an option.
I'm sure that plenty of others on this list have had to deal with this
problem. Any suggestions? How have others dealt with it?
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org