You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jonathan Yom-Tov <jo...@sysaid.com> on 2020/05/12 09:51:01 UTC
seamless restart
I have an application which changes the state of user sessions in lots of
places in the code. Is it possible to do a seamless switch of Tomcat
servers, preserving all sessions?
I know I can use PersistentManager to persist sessions and load them. I can
think of two strategies:
1. Persist sessions periodically. This is more robust as I might not
have control of when the server shuts down.
2. Persist sessions on server shutdown.
The problem with the first approach is that I might lose the latest changes
when the new server comes up. The problem with the second is that I'll have
to lock access to the session until the old server is done saving it, which
may make response times very slow.
Is there a good solution to this that I might have overlooked?
Re: seamless restart
Posted by Mark Eggers <it...@yahoo.com.INVALID>.
Chris,
On 5/12/2020 1:25 PM, Christopher Schultz wrote:
> Mark,
>
> On 5/12/20 16:14, Mark Eggers wrote:
>> Chris,
>
>> On 5/12/2020 12:55 PM, Christopher Schultz wrote:
>>> Jonathan,
>>>
>>> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
>>>> The problem is that my application is running on AWS which
>>>> apparently doesn't support multicasting so I can't use
>>>> Tomcat's DeltaManager.
>>>
>>> The membership-manager is separate from the replication-manager,
>>> so this has nothing to do with e.g. DeltaManager.
>>>
>>> You don't have to use multicast. You can use static membership if
>>> you know your node IP addresses.
>>>
>>> Rémy recently added a cloud membership service that uses
>>> Kubernetes as its default membership service. It looks like he
>>> hasn't written any documentation for it, but it exists in Tomcat
>>> 9 and 10.[1]
>>>
>
>> This sounds interesting. I wonder how this will play using
>> multiple availability zones for high availability. It still won't
>> handle region outages, but there are other approaches for that.
>
> I have no idea. There doesn't seem to me to be any reason why
> Kubernetes could not be used across regions. Maybe you wouldn't be
> able to use AWS-kube and might have to do it yourself. I have zero
> experience with Kubernetes, and zero experience with complex AWS
> deployments.
>
>> I'll read the link you sent, and maybe play with that locally with
>> a Kubernetes setup. If I have questions about the set up, would
>> here or the dev list be the place to ask?
>
> I think here would be better, since the answers will be visible to a
> wider group of people.
>
That sounds reasonable. I know a lot of the devs read this group, so
hopefully (once I get started) there will be some answers.
> I'd love to see a writeup about this, including "how to set up
> Kubernetes from scratch to manage your Tomcat cluster" because I know
> literally nothing practical about it.
That and tying it into a cloud environment should be interesting. I'll
need to think about this before writing stuff on a mailing list, let
alone a document.
I'm just starting my journey through Docker / Kubernetes. I've put
together some simple images locally, and even have a private repository
set up running on Nexus 3. My experience with Kubernetes clusters is
limited to running "canned" environments.
It looks like an interesting road.
. . . just my two cents.
/mde/
>
> -chris
>
>>>> I thought of using one of the Store implementations for
>>>> PersistentManager but that has the issues which I mentioned
>>>> earlier. My aim is to get to the point where I can add or take
>>>> away servers from the cluster without impacting user
>>>> experience.
>>>
>>> See above. Sounds like the cloud membership service is what you
>>> are looking for because it (a) handles dynamic membership and (b)
>>> doesn't use multicast.
>>>
>>>> Ideally all state would be stored in a central location (e.g.
>>>> Redis). But, since this is difficult because of the way the
>>>> application is built I thought of using one server and only
>>>> persisting the sessions when the server goes down. But I still
>>>> have to solve the issues I mentioned.
>>> I would avoid single points of failure if possible. A "central
>>> location" tends to be a single point of failure. Tomcat clustered
>>> with e.g. BackupManager and dynamic membership will (a) achieve
>>> your goals and (b) not require additional products.
>>>
>>> Hope that helps, -chris
>>>
>>> [1]
>>> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina
> /tr
>>>
>>>
> ibes/membership/cloud/CloudMembershipService.java#L34
>>>
>>>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>>>> chris@christopherschultz.net> wrote:
>>>
>>>> Jonathan,
>>>
>>>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>>>>> I have an application which changes the state of user
>>>>>>> sessions in lots of places in the code. Is it possible to
>>>>>>> do a seamless switch of Tomcat servers, preserving all
>>>>>>> sessions?
>>>>>>>
>>>>>>> I know I can use PersistentManager to persist sessions
>>>>>>> and load them. I can think of two strategies:
>>>>>>>
>>>>>>> 1. Persist sessions periodically. This is more robust as
>>>>>>> I might not have control of when the server shuts down.
>>>>>>> 2. Persist sessions on server shutdown.
>>>>>>>
>>>>>>>
>>>>>>> The problem with the first approach is that I might lose
>>>>>>> the latest changes when the new server comes up. The
>>>>>>> problem with the second is that I'll have to lock access
>>>>>>> to the session until the old server is done saving it,
>>>>>>> which may make response times very slow.
>>>>>>>
>>>>>>> Is there a good solution to this that I might have
>>>>>>> overlooked?
>>>
>>>> If you want to solve these problems:
>>>
>>>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date
>>>> (well, as much as possible) 3. No downtime
>>>
>>>> Then you really need a cluster where the sessions are being
>>>> replicated around the cluster.
>>>
>>>> This will solve some other problems as well:
>>>
>>>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>>>> Unexpected downtime (network outage, hardware fault) 6.
>>>> Scaling-out (either manually or automatically)
>>>
>>>> You can do it with as little as two Tomcat instances. If you
>>>> only care about being able to restart your application (and not
>>>> the whole server, for example), then you can even run them
>>>> side-by-side on the same server. You won't get protection
>>>> against OS upgrades and unexpected downtime in that case, but
>>>> you can get familiar with the setup without a whole lot of
>>>> infrastructure.
>>>
>>>> -chris
>>>>>
>>>>> -------------------------------------------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: seamless restart
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 5/12/20 16:14, Mark Eggers wrote:
> Chris,
>
> On 5/12/2020 12:55 PM, Christopher Schultz wrote:
>> Jonathan,
>>
>> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
>>> The problem is that my application is running on AWS which
>>> apparently doesn't support multicasting so I can't use
>>> Tomcat's DeltaManager.
>>
>> The membership-manager is separate from the replication-manager,
>> so this has nothing to do with e.g. DeltaManager.
>>
>> You don't have to use multicast. You can use static membership if
>> you know your node IP addresses.
>>
>> Rémy recently added a cloud membership service that uses
>> Kubernetes as its default membership service. It looks like he
>> hasn't written any documentation for it, but it exists in Tomcat
>> 9 and 10.[1]
>>
>
> This sounds interesting. I wonder how this will play using
> multiple availability zones for high availability. It still won't
> handle region outages, but there are other approaches for that.
I have no idea. There doesn't seem to me to be any reason why
Kubernetes could not be used across regions. Maybe you wouldn't be
able to use AWS-kube and might have to do it yourself. I have zero
experience with Kubernetes, and zero experience with complex AWS
deployments.
> I'll read the link you sent, and maybe play with that locally with
> a Kubernetes setup. If I have questions about the set up, would
> here or the dev list be the place to ask?
I think here would be better, since the answers will be visible to a
wider group of people.
I'd love to see a writeup about this, including "how to set up
Kubernetes from scratch to manage your Tomcat cluster" because I know
literally nothing practical about it.
- -chris
>>> I thought of using one of the Store implementations for
>>> PersistentManager but that has the issues which I mentioned
>>> earlier. My aim is to get to the point where I can add or take
>>> away servers from the cluster without impacting user
>>> experience.
>>
>> See above. Sounds like the cloud membership service is what you
>> are looking for because it (a) handles dynamic membership and (b)
>> doesn't use multicast.
>>
>>> Ideally all state would be stored in a central location (e.g.
>>> Redis). But, since this is difficult because of the way the
>>> application is built I thought of using one server and only
>>> persisting the sessions when the server goes down. But I still
>>> have to solve the issues I mentioned.
>> I would avoid single points of failure if possible. A "central
>> location" tends to be a single point of failure. Tomcat clustered
>> with e.g. BackupManager and dynamic membership will (a) achieve
>> your goals and (b) not require additional products.
>>
>> Hope that helps, -chris
>>
>> [1]
>> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina
/tr
>>
>>
ibes/membership/cloud/CloudMembershipService.java#L34
>>
>>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>>> chris@christopherschultz.net> wrote:
>>
>>> Jonathan,
>>
>>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>>>> I have an application which changes the state of user
>>>>>> sessions in lots of places in the code. Is it possible to
>>>>>> do a seamless switch of Tomcat servers, preserving all
>>>>>> sessions?
>>>>>>
>>>>>> I know I can use PersistentManager to persist sessions
>>>>>> and load them. I can think of two strategies:
>>>>>>
>>>>>> 1. Persist sessions periodically. This is more robust as
>>>>>> I might not have control of when the server shuts down.
>>>>>> 2. Persist sessions on server shutdown.
>>>>>>
>>>>>>
>>>>>> The problem with the first approach is that I might lose
>>>>>> the latest changes when the new server comes up. The
>>>>>> problem with the second is that I'll have to lock access
>>>>>> to the session until the old server is done saving it,
>>>>>> which may make response times very slow.
>>>>>>
>>>>>> Is there a good solution to this that I might have
>>>>>> overlooked?
>>
>>> If you want to solve these problems:
>>
>>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date
>>> (well, as much as possible) 3. No downtime
>>
>>> Then you really need a cluster where the sessions are being
>>> replicated around the cluster.
>>
>>> This will solve some other problems as well:
>>
>>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>>> Unexpected downtime (network outage, hardware fault) 6.
>>> Scaling-out (either manually or automatically)
>>
>>> You can do it with as little as two Tomcat instances. If you
>>> only care about being able to restart your application (and not
>>> the whole server, for example), then you can even run them
>>> side-by-side on the same server. You won't get protection
>>> against OS upgrades and unexpected downtime in that case, but
>>> you can get familiar with the setup without a whole lot of
>>> infrastructure.
>>
>>> -chris
>>>>
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67BlUACgkQHPApP6U8
pFj9EhAAi/CkDkvoZTVPdLHJS9G2UPmrUaO1JjAgKjfy/9NMxJlkcWZmo08roD9h
DeJcDdmvC2dzwuZx7KjoMAvE7TxoNpHurD/ug+/i0hLXDYCUQIbuAQ0EdwZ3rZdQ
VwvBaYQy+wjP48ZML0Foi4Qq+JX+ZbMho/XklyqZXRUx7JeX/P7CyrbUwwPqAqRi
MiI8M0qoioayrbGAkRWQO6LWJL50+01MH8ydNS3O6x3Wr1VkmoB276qj1Jcb8u4P
BRFk+F9yMFy4oBvoH4Qgx8LdoHQc0gBr1oUNZsZceJOxyXrtA+AIhHkU7Nhy2Mf5
6dO7XOiB+8PxK/5mdeqf+1Zi9ZKirqtGhOeGYiMJwkmZa6aOb839j0iRzp+kHpKt
qrU4UsfJO9TgDHswArKpsZBuTYQvmKZSeOsJZBlg9TfQakOfHaAVDCF8TXD2D1Zf
VSOKWdX7OG9mt0fXP8XRPhyO/m4mPEpA7VNIPPc0nJjZ01iVYnFopBcJ1jIo+yEg
1h9TNUNPEZsuAsQWrDHncqJWUfKES2amd/6kOwoCq8+uKHoaFgBGjnI7jrvZFffQ
Ab+wD4epC24Fmw4IBMZ20WcYJxr/SS3xzJKxginvD/QODF67GkoggEZr5c+5kkcL
k29lvV8SqIskVgmTVyxq7FInTiF7LY70hRrsQ+E4mlADMB+243U=
=yo3D
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: seamless restart
Posted by Mark Eggers <it...@yahoo.com.INVALID>.
Chris,
On 5/12/2020 12:55 PM, Christopher Schultz wrote:
> Jonathan,
>
> On 5/12/20 11:20, Jonathan Yom-Tov wrote:
>> The problem is that my application is running on AWS which
>> apparently doesn't support multicasting so I can't use Tomcat's
>> DeltaManager.
>
> The membership-manager is separate from the replication-manager, so
> this has nothing to do with e.g. DeltaManager.
>
> You don't have to use multicast. You can use static membership if you
> know your node IP addresses.
>
> Rémy recently added a cloud membership service that uses Kubernetes as
> its default membership service. It looks like he hasn't written any
> documentation for it, but it exists in Tomcat 9 and 10.[1]
>
This sounds interesting. I wonder how this will play using multiple
availability zones for high availability. It still won't handle region
outages, but there are other approaches for that.
I'll read the link you sent, and maybe play with that locally with a
Kubernetes setup. If I have questions about the set up, would here or
the dev list be the place to ask?
Thanks!
. . . just my two cents
/mde/
>> I thought of using one of the Store implementations for
>> PersistentManager but that has the issues which I mentioned
>> earlier. My aim is to get to the point where I can add or take away
>> servers from the cluster without impacting user experience.
>
> See above. Sounds like the cloud membership service is what you are
> looking for because it (a) handles dynamic membership and (b) doesn't
> use multicast.
>
>> Ideally all state would be stored in a central location (e.g.
>> Redis). But, since this is difficult because of the way the
>> application is built I thought of using one server and only
>> persisting the sessions when the server goes down. But I still have
>> to solve the issues I mentioned.
> I would avoid single points of failure if possible. A "central
> location" tends to be a single point of failure. Tomcat clustered with
> e.g. BackupManager and dynamic membership will (a) achieve your goals
> and (b) not require additional products.
>
> Hope that helps,
> -chris
>
> [1]
> https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/tr
> ibes/membership/cloud/CloudMembershipService.java#L34
>
>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>> chris@christopherschultz.net> wrote:
>
>> Jonathan,
>
>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>>> I have an application which changes the state of user
>>>>> sessions in lots of places in the code. Is it possible to do
>>>>> a seamless switch of Tomcat servers, preserving all
>>>>> sessions?
>>>>>
>>>>> I know I can use PersistentManager to persist sessions and
>>>>> load them. I can think of two strategies:
>>>>>
>>>>> 1. Persist sessions periodically. This is more robust as I
>>>>> might not have control of when the server shuts down. 2.
>>>>> Persist sessions on server shutdown.
>>>>>
>>>>>
>>>>> The problem with the first approach is that I might lose the
>>>>> latest changes when the new server comes up. The problem with
>>>>> the second is that I'll have to lock access to the session
>>>>> until the old server is done saving it, which may make
>>>>> response times very slow.
>>>>>
>>>>> Is there a good solution to this that I might have
>>>>> overlooked?
>
>> If you want to solve these problems:
>
>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date (well, as
>> much as possible) 3. No downtime
>
>> Then you really need a cluster where the sessions are being
>> replicated around the cluster.
>
>> This will solve some other problems as well:
>
>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>> Unexpected downtime (network outage, hardware fault) 6. Scaling-out
>> (either manually or automatically)
>
>> You can do it with as little as two Tomcat instances. If you only
>> care about being able to restart your application (and not the
>> whole server, for example), then you can even run them side-by-side
>> on the same server. You won't get protection against OS upgrades
>> and unexpected downtime in that case, but you can get familiar with
>> the setup without a whole lot of infrastructure.
>
>> -chris
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: seamless restart
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jonathan,
On 5/12/20 11:20, Jonathan Yom-Tov wrote:
> The problem is that my application is running on AWS which
> apparently doesn't support multicasting so I can't use Tomcat's
> DeltaManager.
The membership-manager is separate from the replication-manager, so
this has nothing to do with e.g. DeltaManager.
You don't have to use multicast. You can use static membership if you
know your node IP addresses.
Rémy recently added a cloud membership service that uses Kubernetes as
its default membership service. It looks like he hasn't written any
documentation for it, but it exists in Tomcat 9 and 10.[1]
> I thought of using one of the Store implementations for
> PersistentManager but that has the issues which I mentioned
> earlier. My aim is to get to the point where I can add or take away
> servers from the cluster without impacting user experience.
See above. Sounds like the cloud membership service is what you are
looking for because it (a) handles dynamic membership and (b) doesn't
use multicast.
> Ideally all state would be stored in a central location (e.g.
> Redis). But, since this is difficult because of the way the
> application is built I thought of using one server and only
> persisting the sessions when the server goes down. But I still have
> to solve the issues I mentioned.
I would avoid single points of failure if possible. A "central
location" tends to be a single point of failure. Tomcat clustered with
e.g. BackupManager and dynamic membership will (a) achieve your goals
and (b) not require additional products.
Hope that helps,
- -chris
[1]
https://github.com/apache/tomcat/blob/master/java/org/apache/catalina/tr
ibes/membership/cloud/CloudMembershipService.java#L34
> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Jonathan,
>
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>> I have an application which changes the state of user
>>>> sessions in lots of places in the code. Is it possible to do
>>>> a seamless switch of Tomcat servers, preserving all
>>>> sessions?
>>>>
>>>> I know I can use PersistentManager to persist sessions and
>>>> load them. I can think of two strategies:
>>>>
>>>> 1. Persist sessions periodically. This is more robust as I
>>>> might not have control of when the server shuts down. 2.
>>>> Persist sessions on server shutdown.
>>>>
>>>>
>>>> The problem with the first approach is that I might lose the
>>>> latest changes when the new server comes up. The problem with
>>>> the second is that I'll have to lock access to the session
>>>> until the old server is done saving it, which may make
>>>> response times very slow.
>>>>
>>>> Is there a good solution to this that I might have
>>>> overlooked?
>
> If you want to solve these problems:
>
> 1. Seamless (uninterrupted) restarts 2. Always up-to-date (well, as
> much as possible) 3. No downtime
>
> Then you really need a cluster where the sessions are being
> replicated around the cluster.
>
> This will solve some other problems as well:
>
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
> Unexpected downtime (network outage, hardware fault) 6. Scaling-out
> (either manually or automatically)
>
> You can do it with as little as two Tomcat instances. If you only
> care about being able to restart your application (and not the
> whole server, for example), then you can even run them side-by-side
> on the same server. You won't get protection against OS upgrades
> and unexpected downtime in that case, but you can get familiar with
> the setup without a whole lot of infrastructure.
>
> -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66/x4ACgkQHPApP6U8
pFhd8g//fAo+sX47/md+l9ZxwJnSLWic4c7idAOv1tm3C4PqkQacdfUmaq1jVurc
g9ZMh5NucdM65iubtkXtQrGOyO99kz2ZT9zrz3aFGyF3LY/F0YT3+i8bRtHyDTPp
ATEIJ4Bk+TkoHRRWTzJU3R6SMxa055KLBaEUWKUx8mEaXWNT+/bVtmxigKE0z2Ja
6qI6S1IzAdELRHUd2oVT5IjyBV/b2mjDG7N5YAJGNeCQ2obNrKBuQHdSElcA0Bjn
kXQo+ZHO1pvoMutgV49dvlAeT6u87KnNTPfwJcb0uEO4Eof7NbTswy40lASBRfp2
kRNrdFal/IE9wYUZF8GjVRU1kHOQB0+T8ZeW9hvpCvEsBWKwP74sPHVp71uudW/5
Ggxo5JkoIRvsi+2b6wPhRlQX4YG3awoBoZrGRSDGRaQttdh732hUVbvzSv9b2Z6F
JD0xSy8G5fqPUwouwrF3HIneszrZif1ZoDB6M/Py5i9Fs4F4UxagIEfb59/lTgy+
7GI/fgNV28rjR9QQJXWRp4j8uRreagSDK1Aehfq7cbAPdTpnh+yQlIkeCgYS3A05
wEzFcOlllLb6XW6UBle9GtY72opF/nAZGtToGeGiZvlP+t2x49aBdy6rN21oSyVC
BazreimVfhfS2FrYxYMOJJ8Fj52bSJgIj3dM39vvWpAdnjckg5k=
=8cBm
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: seamless restart
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jonathan,
On 5/12/20 14:19, Jonathan Yom-Tov wrote:
> Thanks Mark. I've tried to use Redisson, it would've been the
> perfect solution for this except for the fact that my session
> object is a deep tree which is mutated in many areas of the code.
> So what happens is that as one request is changing the session
> state another will persist its session to Redis and overwrite the
> session of the first request. I can't think of an easy way to get
> around this.
This will be a problem with everything you try to do.
To make your code more "distributable", you might have to change
things so that you make a local copy (clone?), mutate the clone, then
write it back to the session.
Unless you call session.setAttribute (or similar), Tomcat won't
distribute anything to the other nodes in a cluster. For example:
((Transaction)session.getAttribute("tx")).setStatus(TransactionStaus.COM
PLETE);
Will not cause the cluster to replicate, but this will:
Transaction tx = ((Transaction)session.getAttribute("tx"));
tx.setStatus(TransactionStaus.COMPLETE);
session.setAttribute("tx", tx);
I think you will find many session managers work this way, otherwise
they end up being hideously inefficient by replicating the entire
session across the cluster (or to the session-store) after every request
.
- -chris
> On Tue, May 12, 2020 at 8:39 PM Mark Eggers
> <it...@yahoo.com.invalid> wrote:
>
>> Jonathan,
>>
>> On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
>>> The problem is that my application is running on AWS which
>>> apparently doesn't support multicasting so I can't use Tomcat's
>>> DeltaManager. I thought of using one of the Store
>>> implementations for
>> PersistentManager but
>>> that has the issues which I mentioned earlier. My aim is to get
>>> to the point where I can add or take away servers from the
>>> cluster without impacting user experience. Ideally all state
>>> would be stored in a central location (e.g. Redis). But, since
>>> this is difficult because of the way
>> the
>>> application is built I thought of using one server and only
>>> persisting
>> the
>>> sessions when the server goes down. But I still have to solve
>>> the issues
>> I
>>> mentioned.
>>>
>>>
>>>
>>>
>>> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
>>> chris@christopherschultz.net> wrote:
>>>
>>> Jonathan,
>>>
>>> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>>>> I have an application which changes the state of user
>>>>>> sessions in lots of places in the code. Is it possible to
>>>>>> do a seamless switch of Tomcat servers, preserving all
>>>>>> sessions?
>>>>>>
>>>>>> I know I can use PersistentManager to persist sessions
>>>>>> and load them. I can think of two strategies:
>>>>>>
>>>>>> 1. Persist sessions periodically. This is more robust as
>>>>>> I might not have control of when the server shuts down.
>>>>>> 2. Persist sessions on server shutdown.
>>>>>>
>>>>>>
>>>>>> The problem with the first approach is that I might lose
>>>>>> the latest changes when the new server comes up. The
>>>>>> problem with the second is that I'll have to lock access
>>>>>> to the session until the old server is done saving it,
>>>>>> which may make response times very slow.
>>>>>>
>>>>>> Is there a good solution to this that I might have
>>>>>> overlooked?
>>>
>>> If you want to solve these problems:
>>>
>>> 1. Seamless (uninterrupted) restarts 2. Always up-to-date
>>> (well, as much as possible) 3. No downtime
>>>
>>> Then you really need a cluster where the sessions are being
>>> replicated around the cluster.
>>>
>>> This will solve some other problems as well:
>>>
>>> 4. Expected downtime (e.g. OS/Tomcat/application upgrade) 5.
>>> Unexpected downtime (network outage, hardware fault) 6.
>>> Scaling-out (either manually or automatically)
>>>
>>> You can do it with as little as two Tomcat instances. If you
>>> only care about being able to restart your application (and not
>>> the whole server, for example), then you can even run them
>>> side-by-side on the same server. You won't get protection
>>> against OS upgrades and unexpected downtime in that case, but
>>> you can get familiar with the setup without a whole lot of
>>> infrastructure.
>>>
>>> -chris
>>
>> Could you use the RedissonSessionManager and an AWS - distributed
>> Redis server?
>>
>> You could put all of your Tomcat servers in an elastic group, and
>> let AWS manage that.
>>
>> The real problem with this approach is deployment. How do you
>> deploy across an elastic group of Tomcat servers when you may not
>> know the IP addresses of the servers or how many you have?
>>
>> I can think of some really kludgy ways to do this with S3 and
>> AWS events, but I've not worked out the details.
>>
>> Another way to approach this is to run Docker on AWS (along with
>> Redis), and then deploy a new version by deploying a new Docker
>> image in a rolling fashion.
>>
>> If your session interface changes a lot, that could create
>> issues.
>>
>> That's one of the advantages of using versioned deployment
>> (app.war##nnn) with a cluster. Old apps stay around until the
>> session expires, while new sessions get the new version.
>>
>> Maybe -- just thinking out loud -- you could use an elastic
>> group, AWS events, Redis (RedisSessionManager), and numbered WAR
>> files to simulate a Tomcat cluster.
>>
>> Another question: Is the database-backed session manager provided
>> with Tomcat slow? You could use that instead of the third party
>> RedissonSessionManager.
>>
>> You should be able to test everything but the deployment locally.
>> Just run a Docker implementation on your development machine, and
>> then test either RedissonSessionManager or the JDBC backed
>> session store. Docker will (can) be set up to mimic AWS elastic
>> group behavior (expansion / contraction of containers), so the
>> only question will be updates.
>>
>> Use something like JMeter to test sessions and hammer your
>> Docker cluster. By default, Docker routes every request to a new
>> container in a multi-container group. You'll know really quickly
>> if distributed sessions aren't working.
>>
>> I need to get back to this for $work, but I've been getting
>> yanked around a bit. Hopefully, I'll be able to start testing all
>> of these ideas in the next month or so.
>>
>> . . . just my two cents /mde/
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl67APoACgkQHPApP6U8
pFiOABAAyutfg+Yhy/Zn22TMhIvlLWhx8sHNOxXphJNbXugJ1gZmC8Ab+DCBr1N/
+QZVzEcVOj80QdL5iQ3yeKxp5gEmcplYD9A9sIOXzUdTc6tRHjasuu8Y0TALZ/13
bUYQEQKx5ZuFLPCIP91QtdgV2GcRJALbRC6MCi95LW0RO6LR1Ge1FoO+R+Y5EsUX
ci4NesPm6HPoJOROm7H5daeS12Wq4GqZOMZKpS14ouLb01bo1vr7fUrDmQxOTXKL
zs473g/pyiPUxlom2rzVHgU5bNiU0rQcTpMK6PO0YSxRs36WwA59vU5HNYJRyoPQ
MG3lbwHVmDS6MhvcuAJVWIOsEGjxHCfmAN5tysbfmsAv6DNl7dPM0R8WYpVppDiz
Q8lfglbP2VRY5KHJai5QbdPxg+iyAdvcriImJA16s3XpW4X/B8x5uswwlvpOdCOA
OnsXIB9BlIJ++HhqVx7JYLrM9ygUH1M5H9/9rlYNwhSRO2X/DXQH8cA8IX6siMuA
BqfKdMx0kdmxVZBodT6x7wzDBv1e4zgdl4UcWQF2X7tXzgC7+9kkAyQf8TFhG7zr
ZKZQiwZJ+iNr+GwRpfvdsn+jiG6K/zGtFWHUGYnyiifokFhHAfdfE9W0p0gDKKzM
gqEtLs/NN880n9FlpJ3PNajjY13CLgKHC1T+ORiHg/FGXa11TNc=
=RTC7
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: seamless restart
Posted by Jonathan Yom-Tov <jo...@sysaid.com>.
Thanks Mark. I've tried to use Redisson, it would've been the perfect
solution for this except for the fact that my session object is a deep tree
which is mutated in many areas of the code. So what happens is that as one
request is changing the session state another will persist its session to
Redis and overwrite the session of the first request. I can't think of an
easy way to get around this.
On Tue, May 12, 2020 at 8:39 PM Mark Eggers <it...@yahoo.com.invalid>
wrote:
> Jonathan,
>
> On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
> > The problem is that my application is running on AWS which apparently
> > doesn't support multicasting so I can't use Tomcat's DeltaManager. I
> > thought of using one of the Store implementations for
> PersistentManager but
> > that has the issues which I mentioned earlier. My aim is to get to the
> > point where I can add or take away servers from the cluster without
> > impacting user experience. Ideally all state would be stored in a central
> > location (e.g. Redis). But, since this is difficult because of the way
> the
> > application is built I thought of using one server and only persisting
> the
> > sessions when the server goes down. But I still have to solve the issues
> I
> > mentioned.
> >
> >
> >
> >
> > On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> > chris@christopherschultz.net> wrote:
> >
> > Jonathan,
> >
> > On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> >>>> I have an application which changes the state of user sessions in
> >>>> lots of places in the code. Is it possible to do a seamless switch
> >>>> of Tomcat servers, preserving all sessions?
> >>>>
> >>>> I know I can use PersistentManager to persist sessions and load
> >>>> them. I can think of two strategies:
> >>>>
> >>>> 1. Persist sessions periodically. This is more robust as I might
> >>>> not have control of when the server shuts down. 2. Persist sessions
> >>>> on server shutdown.
> >>>>
> >>>>
> >>>> The problem with the first approach is that I might lose the latest
> >>>> changes when the new server comes up. The problem with the second
> >>>> is that I'll have to lock access to the session until the old
> >>>> server is done saving it, which may make response times very slow.
> >>>>
> >>>> Is there a good solution to this that I might have overlooked?
> >
> > If you want to solve these problems:
> >
> > 1. Seamless (uninterrupted) restarts
> > 2. Always up-to-date (well, as much as possible)
> > 3. No downtime
> >
> > Then you really need a cluster where the sessions are being replicated
> > around the cluster.
> >
> > This will solve some other problems as well:
> >
> > 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> > 5. Unexpected downtime (network outage, hardware fault)
> > 6. Scaling-out (either manually or automatically)
> >
> > You can do it with as little as two Tomcat instances. If you only care
> > about being able to restart your application (and not the whole
> > server, for example), then you can even run them side-by-side on the
> > same server. You won't get protection against OS upgrades and
> > unexpected downtime in that case, but you can get familiar with the
> > setup without a whole lot of infrastructure.
> >
> > -chris
>
> Could you use the RedissonSessionManager and an AWS - distributed Redis
> server?
>
> You could put all of your Tomcat servers in an elastic group, and let
> AWS manage that.
>
> The real problem with this approach is deployment. How do you deploy
> across an elastic group of Tomcat servers when you may not know the IP
> addresses of the servers or how many you have?
>
> I can think of some really kludgy ways to do this with S3 and AWS
> events, but I've not worked out the details.
>
> Another way to approach this is to run Docker on AWS (along with Redis),
> and then deploy a new version by deploying a new Docker image in a
> rolling fashion.
>
> If your session interface changes a lot, that could create issues.
>
> That's one of the advantages of using versioned deployment
> (app.war##nnn) with a cluster. Old apps stay around until the session
> expires, while new sessions get the new version.
>
> Maybe -- just thinking out loud -- you could use an elastic group, AWS
> events, Redis (RedisSessionManager), and numbered WAR files to simulate
> a Tomcat cluster.
>
> Another question: Is the database-backed session manager provided with
> Tomcat slow? You could use that instead of the third party
> RedissonSessionManager.
>
> You should be able to test everything but the deployment locally. Just
> run a Docker implementation on your development machine, and then test
> either RedissonSessionManager or the JDBC backed session store. Docker
> will (can) be set up to mimic AWS elastic group behavior (expansion /
> contraction of containers), so the only question will be updates.
>
> Use something like JMeter to test sessions and hammer your Docker
> cluster. By default, Docker routes every request to a new container in a
> multi-container group. You'll know really quickly if distributed
> sessions aren't working.
>
> I need to get back to this for $work, but I've been getting yanked
> around a bit. Hopefully, I'll be able to start testing all of these
> ideas in the next month or so.
>
> . . . just my two cents
> /mde/
>
>
Re: seamless restart
Posted by Mark Eggers <it...@yahoo.com.INVALID>.
Jonathan,
On 5/12/2020 8:20 AM, Jonathan Yom-Tov wrote:
> The problem is that my application is running on AWS which apparently
> doesn't support multicasting so I can't use Tomcat's DeltaManager. I
> thought of using one of the Store implementations for
PersistentManager but
> that has the issues which I mentioned earlier. My aim is to get to the
> point where I can add or take away servers from the cluster without
> impacting user experience. Ideally all state would be stored in a central
> location (e.g. Redis). But, since this is difficult because of the way the
> application is built I thought of using one server and only persisting the
> sessions when the server goes down. But I still have to solve the issues I
> mentioned.
>
>
>
>
> On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Jonathan,
>
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
>>>> I have an application which changes the state of user sessions in
>>>> lots of places in the code. Is it possible to do a seamless switch
>>>> of Tomcat servers, preserving all sessions?
>>>>
>>>> I know I can use PersistentManager to persist sessions and load
>>>> them. I can think of two strategies:
>>>>
>>>> 1. Persist sessions periodically. This is more robust as I might
>>>> not have control of when the server shuts down. 2. Persist sessions
>>>> on server shutdown.
>>>>
>>>>
>>>> The problem with the first approach is that I might lose the latest
>>>> changes when the new server comes up. The problem with the second
>>>> is that I'll have to lock access to the session until the old
>>>> server is done saving it, which may make response times very slow.
>>>>
>>>> Is there a good solution to this that I might have overlooked?
>
> If you want to solve these problems:
>
> 1. Seamless (uninterrupted) restarts
> 2. Always up-to-date (well, as much as possible)
> 3. No downtime
>
> Then you really need a cluster where the sessions are being replicated
> around the cluster.
>
> This will solve some other problems as well:
>
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> 5. Unexpected downtime (network outage, hardware fault)
> 6. Scaling-out (either manually or automatically)
>
> You can do it with as little as two Tomcat instances. If you only care
> about being able to restart your application (and not the whole
> server, for example), then you can even run them side-by-side on the
> same server. You won't get protection against OS upgrades and
> unexpected downtime in that case, but you can get familiar with the
> setup without a whole lot of infrastructure.
>
> -chris
Could you use the RedissonSessionManager and an AWS - distributed Redis
server?
You could put all of your Tomcat servers in an elastic group, and let
AWS manage that.
The real problem with this approach is deployment. How do you deploy
across an elastic group of Tomcat servers when you may not know the IP
addresses of the servers or how many you have?
I can think of some really kludgy ways to do this with S3 and AWS
events, but I've not worked out the details.
Another way to approach this is to run Docker on AWS (along with Redis),
and then deploy a new version by deploying a new Docker image in a
rolling fashion.
If your session interface changes a lot, that could create issues.
That's one of the advantages of using versioned deployment
(app.war##nnn) with a cluster. Old apps stay around until the session
expires, while new sessions get the new version.
Maybe -- just thinking out loud -- you could use an elastic group, AWS
events, Redis (RedisSessionManager), and numbered WAR files to simulate
a Tomcat cluster.
Another question: Is the database-backed session manager provided with
Tomcat slow? You could use that instead of the third party
RedissonSessionManager.
You should be able to test everything but the deployment locally. Just
run a Docker implementation on your development machine, and then test
either RedissonSessionManager or the JDBC backed session store. Docker
will (can) be set up to mimic AWS elastic group behavior (expansion /
contraction of containers), so the only question will be updates.
Use something like JMeter to test sessions and hammer your Docker
cluster. By default, Docker routes every request to a new container in a
multi-container group. You'll know really quickly if distributed
sessions aren't working.
I need to get back to this for $work, but I've been getting yanked
around a bit. Hopefully, I'll be able to start testing all of these
ideas in the next month or so.
. . . just my two cents
/mde/
Re: seamless restart
Posted by Jonathan Yom-Tov <jo...@sysaid.com>.
The problem is that my application is running on AWS which apparently
doesn't support multicasting so I can't use Tomcat's DeltaManager. I
thought of using one of the Store implementations for PersistentManager but
that has the issues which I mentioned earlier. My aim is to get to the
point where I can add or take away servers from the cluster without
impacting user experience. Ideally all state would be stored in a central
location (e.g. Redis). But, since this is difficult because of the way the
application is built I thought of using one server and only persisting the
sessions when the server goes down. But I still have to solve the issues I
mentioned.
On Tue, May 12, 2020 at 6:06 PM Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Jonathan,
>
> On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> > I have an application which changes the state of user sessions in
> > lots of places in the code. Is it possible to do a seamless switch
> > of Tomcat servers, preserving all sessions?
> >
> > I know I can use PersistentManager to persist sessions and load
> > them. I can think of two strategies:
> >
> > 1. Persist sessions periodically. This is more robust as I might
> > not have control of when the server shuts down. 2. Persist sessions
> > on server shutdown.
> >
> >
> > The problem with the first approach is that I might lose the latest
> > changes when the new server comes up. The problem with the second
> > is that I'll have to lock access to the session until the old
> > server is done saving it, which may make response times very slow.
> >
> > Is there a good solution to this that I might have overlooked?
>
> If you want to solve these problems:
>
> 1. Seamless (uninterrupted) restarts
> 2. Always up-to-date (well, as much as possible)
> 3. No downtime
>
> Then you really need a cluster where the sessions are being replicated
> around the cluster.
>
> This will solve some other problems as well:
>
> 4. Expected downtime (e.g. OS/Tomcat/application upgrade)
> 5. Unexpected downtime (network outage, hardware fault)
> 6. Scaling-out (either manually or automatically)
>
> You can do it with as little as two Tomcat instances. If you only care
> about being able to restart your application (and not the whole
> server, for example), then you can even run them side-by-side on the
> same server. You won't get protection against OS upgrades and
> unexpected downtime in that case, but you can get familiar with the
> setup without a whole lot of infrastructure.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66rKgACgkQHPApP6U8
> pFjsChAAtKdo2Ow+X+Lb0JQLAOxijryxlsRtO3rvdRPJLUxVG4ttVGXuJ86cPX9g
> uRoyOrtgSw3N7mPMgL51B39Sp8GtB6zcjAiB4CQzOTXA3FiRN99Jna6dEVpCUcQx
> xAUBNuJh6Q5N/l1tpLYIMgKXEmhGcexplNTEVl8IAb5tppNy1LtjUVMG3C4NolI1
> eKUVqNhtSmclBH9aDC94jlWLVH+OiKJhAfc965oCRv4E3vIcj5HGoKfK65iqFYPx
> hzOMUzvJfsHBfNMo0UyKvRiDIFyBUgeUEIAJZWAJojhTSHymYpQozYBDIjCfNHYa
> II6NYhYb9OuiwV9PEXFN2yZejBsgJvZzlbooWOg2z4nMmCItaTiUhOMyKNIY5wlz
> iXnh85yx/GMRSsiQrTD6X4cAwCsXusHwoF/WC0JncYr7hZw/mqwcXDnMhUAccOCc
> Z1l2x53yoVSH+RyoGgdG/DSWqLHZfiK/xr8UOtRbMD1WNfGOPo+GhpJBiSNSBeeS
> LMNiQbvZEcTKOsxUFgUDPwB62kbDyLnt765tgrSOWubBIaHXBctTNLtv9sjp6ARG
> DnNTmAtdk89UuhDA63iimDxx9jxDKsWKPz8Srf6RS7z8yUrlvFkfKQcIEaqHSLTr
> LnbrTHK2k92bJonJua5hDcuTgU7axEBlWmXxChyZWmYm2r1R76M=
> =suSa
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
[image: SysAid Technologies]
<http://www.sysaid.com/?utm_source=signature&utm_medium=email&utm_campaign=sysaid-logo>
Jonathan Yom-Tov
Senior Architect
jonathan.yomtov@sysaid.com
Phone (IL): +972 (3) 533-3675 Ext. 932
[image: SysAid Technologies]
<https://www.sysaid.com/?utm_source=signature&utm_medium=email&utm_campaign=sysaid-logo-icon>
[image: SysAid on Facebook] <https://www.facebook.com/SysAidIT> [image:
SysAid on Twitter] <https://twitter.com/sysaid> [image: SysAid on
Linked-in] <https://www.linkedin.com/company/sysaid-technologies-ltd> [image:
SysAid on YouTube] <https://www.youtube.com/user/SysAidIT> [image: SysAid
on Instagram] <https://www.instagram.com/sysaid_technologies/>
[image: Banner] <https://www.sysaid.com/sig-link>
Re: seamless restart
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jonathan,
On 5/12/20 05:51, Jonathan Yom-Tov wrote:
> I have an application which changes the state of user sessions in
> lots of places in the code. Is it possible to do a seamless switch
> of Tomcat servers, preserving all sessions?
>
> I know I can use PersistentManager to persist sessions and load
> them. I can think of two strategies:
>
> 1. Persist sessions periodically. This is more robust as I might
> not have control of when the server shuts down. 2. Persist sessions
> on server shutdown.
>
>
> The problem with the first approach is that I might lose the latest
> changes when the new server comes up. The problem with the second
> is that I'll have to lock access to the session until the old
> server is done saving it, which may make response times very slow.
>
> Is there a good solution to this that I might have overlooked?
If you want to solve these problems:
1. Seamless (uninterrupted) restarts
2. Always up-to-date (well, as much as possible)
3. No downtime
Then you really need a cluster where the sessions are being replicated
around the cluster.
This will solve some other problems as well:
4. Expected downtime (e.g. OS/Tomcat/application upgrade)
5. Unexpected downtime (network outage, hardware fault)
6. Scaling-out (either manually or automatically)
You can do it with as little as two Tomcat instances. If you only care
about being able to restart your application (and not the whole
server, for example), then you can even run them side-by-side on the
same server. You won't get protection against OS upgrades and
unexpected downtime in that case, but you can get familiar with the
setup without a whole lot of infrastructure.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl66rKgACgkQHPApP6U8
pFjsChAAtKdo2Ow+X+Lb0JQLAOxijryxlsRtO3rvdRPJLUxVG4ttVGXuJ86cPX9g
uRoyOrtgSw3N7mPMgL51B39Sp8GtB6zcjAiB4CQzOTXA3FiRN99Jna6dEVpCUcQx
xAUBNuJh6Q5N/l1tpLYIMgKXEmhGcexplNTEVl8IAb5tppNy1LtjUVMG3C4NolI1
eKUVqNhtSmclBH9aDC94jlWLVH+OiKJhAfc965oCRv4E3vIcj5HGoKfK65iqFYPx
hzOMUzvJfsHBfNMo0UyKvRiDIFyBUgeUEIAJZWAJojhTSHymYpQozYBDIjCfNHYa
II6NYhYb9OuiwV9PEXFN2yZejBsgJvZzlbooWOg2z4nMmCItaTiUhOMyKNIY5wlz
iXnh85yx/GMRSsiQrTD6X4cAwCsXusHwoF/WC0JncYr7hZw/mqwcXDnMhUAccOCc
Z1l2x53yoVSH+RyoGgdG/DSWqLHZfiK/xr8UOtRbMD1WNfGOPo+GhpJBiSNSBeeS
LMNiQbvZEcTKOsxUFgUDPwB62kbDyLnt765tgrSOWubBIaHXBctTNLtv9sjp6ARG
DnNTmAtdk89UuhDA63iimDxx9jxDKsWKPz8Srf6RS7z8yUrlvFkfKQcIEaqHSLTr
LnbrTHK2k92bJonJua5hDcuTgU7axEBlWmXxChyZWmYm2r1R76M=
=suSa
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org