You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Aristedes Maniatis <am...@apache.org> on 2012/02/23 07:51:06 UTC
parallel deployment: multiple applications responding
We've been using parallel deployment for some time now with tomcat 7.0.25. For the most part the parallel deployment is a really nice idea, particularly because we don't have sessions serialised and clustered across all running instances.
However, we've had mysterious problems for some time now where fixes we've applied to the application don't seem to work. We added some logging and have confirmed that some requests are still being served by the OLD instances of the application which are still deployed in the tomcat container but have an older version and therefore should not be responding. We might have two apps like this (using a format of YYMMDD and a two digit sequence):
enrol##12022203.war
enrol##12021503.war
We would expect that requests would be served from only the newer application, but we find that requests continue to be served from the old, even though all the sessions to the old application are long dead.
Just to confirm things we have another application which does not use sessions at all. It also has the same problem: requests are being served by the wrong application.
In Tomcat Manager, the old application is still marked as running, but we thought this is just how it appears in the UI. Clearly there is something not right here.
1. Is this a known issue?
2. Is there some way to get the old application to fully undeploy as soon as it has no live sessions? We have been thinking about writing an external application to poll using JMX and do this, but that's quite a bit of work. It would be nicer if this happened inside tomcat itself.
3. Is there some resource we might be hanging onto which is preventing the old application from properly stopping?
4. Should the tomcat manager show the older apps as still running or should they be shown as stopped?
Cheers
Ari Maniatis
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: parallel deployment: multiple applications responding
Posted by Pid <pi...@pidster.com>.
On 27/02/2012 22:23, Aristedes Maniatis wrote:
> On 28/02/12 12:14 AM, Christopher Schultz wrote:
> Chuck,
>
> On 2/26/12 11:29 PM, Caldarale, Charles R wrote:
>>>>> From: Aristedes Maniatis [mailto:amaniatis@apache.org] Subject:
>>>>> Re: parallel deployment: multiple applications responding
>>>>
>>>>> What happens if our application defines a static class or other
>>>>> resource?
>>>>
>>>> Not sure what you mean by "static class", unless you're referring
>>>> to an inner class. Regardless, each parallel deployment uses a
>>>> separate classloader, so the webapp instances cannot interfere
>>>> with each other.
>
> I'm sure he means a class with a static method that returns a shared
> object (aka singleton, etc.).
>
>> Yes of course. I mispoke earlier.
>
>
>>>> However, if you've placed classes in a shared library (e.g.,
>>>> Tomcat's lib directory), anything in there will be shared by all
>>>> webapps. You must be very, very careful when utilizing shared
>>>> classes, since that can easily lead to unexpected data leakage
>>>> between webapps (not to mention often making it impossible to
>>>> undeploy them).
>
> Yes, but I would be very surprised if one could cause a different
> webapp to service a request by doing such a thing. What you might be
> able to do is get the request *logged* to the wrong webapp, if you
> were fetching references (say, to the ServletContext) and caching them
> in the (shared) class, then calling ServletContext.log on them.
>
> Aristedes, can you describe in a bit more detail the kinds of things
> you are doing, here?
>
> Also, what kind of logging are you using that suggests your requests
> are being handled by the wrong webapp? What does JMX have to say about
> the number of active sessions? (You said that the 'manager' webapp
> shows a status for the webapp, so you are probably using that session
> count in your reports, here, but I just wanted to check).
>
>
>> I'll focus on trying to get some better logging of the session into the
>> log output and see if we can narrow down the symptoms. Yes, I am using
>> JMX and the Manager app to identify when there are no remaining
>> sessions, and when I therefore should expect to see no further activity
>> at all from the older application. I'll try and get some better
>> information around that by adding logging as you suggest.
>
>> The logging that we did originally to discover this problem was simple:
>
>> * sessionless application
>> * we changed the log.properties for log4j in the new app
>> * some hours after we deployed, the old-style log entries were still
>> being written, mixed in with the new ones
Aristedes,
Did you manage to resolve the problem or narrow down on what is actually
happening?
p
Re: parallel deployment: multiple applications responding
Posted by Aristedes Maniatis <am...@apache.org>.
On 28/02/12 12:14 AM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Chuck,
>
> On 2/26/12 11:29 PM, Caldarale, Charles R wrote:
>>> From: Aristedes Maniatis [mailto:amaniatis@apache.org] Subject:
>>> Re: parallel deployment: multiple applications responding
>>
>>> What happens if our application defines a static class or other
>>> resource?
>>
>> Not sure what you mean by "static class", unless you're referring
>> to an inner class. Regardless, each parallel deployment uses a
>> separate classloader, so the webapp instances cannot interfere
>> with each other.
>
> I'm sure he means a class with a static method that returns a shared
> object (aka singleton, etc.).
Yes of course. I mispoke earlier.
>> However, if you've placed classes in a shared library (e.g.,
>> Tomcat's lib directory), anything in there will be shared by all
>> webapps. You must be very, very careful when utilizing shared
>> classes, since that can easily lead to unexpected data leakage
>> between webapps (not to mention often making it impossible to
>> undeploy them).
>
> Yes, but I would be very surprised if one could cause a different
> webapp to service a request by doing such a thing. What you might be
> able to do is get the request *logged* to the wrong webapp, if you
> were fetching references (say, to the ServletContext) and caching them
> in the (shared) class, then calling ServletContext.log on them.
>
> Aristedes, can you describe in a bit more detail the kinds of things
> you are doing, here?
>
> Also, what kind of logging are you using that suggests your requests
> are being handled by the wrong webapp? What does JMX have to say about
> the number of active sessions? (You said that the 'manager' webapp
> shows a status for the webapp, so you are probably using that session
> count in your reports, here, but I just wanted to check).
I'll focus on trying to get some better logging of the session into the log output and see if we can narrow down the symptoms. Yes, I am using JMX and the Manager app to identify when there are no remaining sessions, and when I therefore should expect to see no further activity at all from the older application. I'll try and get some better information around that by adding logging as you suggest.
The logging that we did originally to discover this problem was simple:
* sessionless application
* we changed the log.properties for log4j in the new app
* some hours after we deployed, the old-style log entries were still being written, mixed in with the new ones
Thanks for your suggestions, I'll investigate a few more things and get better information. At least I know this isn't some simple newbie known issue.
Cheers
Ari
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk9LgbUACgkQ9CaO5/Lv0PBYoACfSFRbddQN1BNkBT5Q4GzhP2zA
> o4MAnRZVmV0mhFOj7Wu94JX76nbV9DFB
> =ugoA
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: parallel deployment: multiple applications responding
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chuck,
On 2/26/12 11:29 PM, Caldarale, Charles R wrote:
>> From: Aristedes Maniatis [mailto:amaniatis@apache.org] Subject:
>> Re: parallel deployment: multiple applications responding
>
>> What happens if our application defines a static class or other
>> resource?
>
> Not sure what you mean by "static class", unless you're referring
> to an inner class. Regardless, each parallel deployment uses a
> separate classloader, so the webapp instances cannot interfere
> with each other.
I'm sure he means a class with a static method that returns a shared
object (aka singleton, etc.).
> However, if you've placed classes in a shared library (e.g.,
> Tomcat's lib directory), anything in there will be shared by all
> webapps. You must be very, very careful when utilizing shared
> classes, since that can easily lead to unexpected data leakage
> between webapps (not to mention often making it impossible to
> undeploy them).
Yes, but I would be very surprised if one could cause a different
webapp to service a request by doing such a thing. What you might be
able to do is get the request *logged* to the wrong webapp, if you
were fetching references (say, to the ServletContext) and caching them
in the (shared) class, then calling ServletContext.log on them.
Aristedes, can you describe in a bit more detail the kinds of things
you are doing, here?
Also, what kind of logging are you using that suggests your requests
are being handled by the wrong webapp? What does JMX have to say about
the number of active sessions? (You said that the 'manager' webapp
shows a status for the webapp, so you are probably using that session
count in your reports, here, but I just wanted to check).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9LgbUACgkQ9CaO5/Lv0PBYoACfSFRbddQN1BNkBT5Q4GzhP2zA
o4MAnRZVmV0mhFOj7Wu94JX76nbV9DFB
=ugoA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: parallel deployment: multiple applications responding
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Aristedes Maniatis [mailto:amaniatis@apache.org]
> Subject: Re: parallel deployment: multiple applications responding
> What happens if our application defines a static class or
> other resource?
Not sure what you mean by "static class", unless you're referring to an inner class. Regardless, each parallel deployment uses a separate classloader, so the webapp instances cannot interfere with each other.
However, if you've placed classes in a shared library (e.g., Tomcat's lib directory), anything in there will be shared by all webapps. You must be very, very careful when utilizing shared classes, since that can easily lead to unexpected data leakage between webapps (not to mention often making it impossible to undeploy them).
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
Re: parallel deployment: multiple applications responding
Posted by Aristedes Maniatis <am...@apache.org>.
On 27/02/12 2:12 PM, Christopher Schultz wrote:
>> It would be nice if it showed as "disabled" (to use the mod_jk
>> terminology for an instance which is running but gets no new
>> sessions).
>
> It's just a coincidence that no new sessions are being created --
> because sessionless requests /should/ be sent to the new version. The
> old version simply isn't used any more... it's not actually in any
> different state than the new version.
Now this is interesting. What happens if our application defines a static class or other resource? Would both the new and old code be simultaneously loaded in the same JVM at the same time? Is there some way we might have been able to create an object in memory which is causing the requests to be handled by the wrong application? We are doing a few tricky things with parsing incoming requests which isn't a standard Tapestry way of working.
Tomcat doesn't show any memory leaks when we press that button in the Tomcat Manager, but perhaps there is something in our code which confuses the code from the old and new applications? Is that even possible?
The feature request for the auto-shutdown issue is here: https://issues.apache.org/bugzilla/show_bug.cgi?id=52777 I took the liberty of taking your notes in this list into the task so that the ideas are not lost.
Cheers
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: parallel deployment: multiple applications responding
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Aristedes,
On 2/23/12 4:29 AM, Aristedes Maniatis wrote:
> Given that we've definitely seen this happen with our sessionless
> application, I'm not sure that will help us much. For our other
> apps which have sessions, what happens if the incoming cookie
> references a session which has expired? Will the connection be
> simply handled by the new application?
Yes.
> Is there some chance that the old application then creates a new
> valid session for that connection?
I suppose there is some chance (there could be a bug) bu I suspect
something else is going on.
Is it possible that some component you aren't expecting is in fact
creating a session that is otherwise unused? Adding the session id to
the logger would certainly shed some light on that.
> What we did do is add a version to our log4j logger so that we
> could see which application was handling requests. We got quite a
> mixture of requests to the older sessionless application.
That's interesting. Please confirm that the requests do not have any
session id associated with them.
> Is there a task I can follow? I was unable to find one.
If you'd like, you can file an enhancement request in Bugzilla. I
wasn't able to find anything anything that already existed.
https://issues.apache.org/bugzilla
>>> We have been thinking about writing an external application to
>>> poll using JMX and do this, but that's quite a bit of work. It
>>> would be nicer if this happened inside tomcat itself.
>>
>> Agreed. Patches welcome.
>
> Sure. I understand, but I would not know where to begin with the
> tomcat codebase. I haven't even tried to read the source at this
> point. I assume we'd need to hook into some listener that detects
> when sessions are terminated and tell when they reach zero. That is
> maybe one line of code in the right place... or not :-)
I haven't looked at the code closely enough to know if this would all
work, but it seems reasonable that you could do the following:
1. Modify the parallel deployment code to register an MBean
NotificationListener that filters for useful events (such as expiring
session notifications on the outgoing webapp).
2. When the listener receives a notification, check the current state
(e.g. session count=0; or, I suppose you could make this a part of
your filter in step #1). If session count = 0, start a new thread that
stops the outgoing webapp and de-registers the listener.
Or.
1. Install a SessionEventListener that counts-down the sessions
(you'll have to get the count from JMX, I suppose) until they equal 0,
then starts a new thread that ... etc.
Or.
1. Install a Timer thread that polls at intervals (1 minute?) to see
if all the sessions are dead and then starts a thread ... etc.
The first idea seems the cleanest, though Tomcat might not actually
fire MBean events for things like session count changing. Again, I
haven't checked the code myself, yet.
>>> 3. Is there some resource we might be hanging onto which is
>>> preventing the old application from properly stopping?
>>
>> Maybe. But that would be a separate issue.
>
> I am just looking for solutions within the particular stack that we
> are using (tapestry/cayenne) to see if there was some reason why
> tomcat wasn't fully letting go of the older application.
If there are pending sessions, Tomcat shouldn't terminate that webapp.
As previously mentioned, Tomcat will not (yet) auto-undeploy the old
version when all sessions have terminated.
>>> 4. Should the tomcat manager show the older apps as still
>>> running or should they be shown as stopped?
>>
>> running.
>
> It would be nice if it showed as "disabled" (to use the mod_jk
> terminology for an instance which is running but gets no new
> sessions).
It's just a coincidence that no new sessions are being created --
because sessionless requests /should/ be sent to the new version. The
old version simply isn't used any more... it's not actually in any
different state than the new version.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk9K9JQACgkQ9CaO5/Lv0PAsDACgmH786rjCWgaQgCWy3L2HjoMp
0w4An12U6AuWEytXy2qzwAMWqsWG6vSQ
=LeiA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: parallel deployment: multiple applications responding
Posted by Aristedes Maniatis <am...@apache.org>.
On 23/02/12 7:24 PM, Mark Thomas wrote:
> On 23/02/2012 06:51, Aristedes Maniatis wrote:
>> We've been using parallel deployment for some time now with tomcat
>> 7.0.25. For the most part the parallel deployment is a really nice idea,
>> particularly because we don't have sessions serialised and clustered
>> across all running instances.
>>
>> However, we've had mysterious problems for some time now where fixes
>> we've applied to the application don't seem to work. We added some
>> logging and have confirmed that some requests are still being served by
>> the OLD instances of the application which are still deployed in the
>> tomcat container but have an older version and therefore should not be
>> responding. We might have two apps like this (using a format of YYMMDD
>> and a two digit sequence):
>>
>> enrol##12022203.war
>> enrol##12021503.war
>>
>> We would expect that requests would be served from only the newer
>> application, but we find that requests continue to be served from the
>> old, even though all the sessions to the old application are long dead.
>>
>> Just to confirm things we have another application which does not use
>> sessions at all. It also has the same problem: requests are being served
>> by the wrong application.
>>
>>
>> In Tomcat Manager, the old application is still marked as running, but
>> we thought this is just how it appears in the UI. Clearly there is
>> something not right here.
>>
>>
>> 1. Is this a known issue?
>
> We'd need to understand the root cause to answer that. The short version
> is that requests will only be routed to the old version if a request
> includes a session ID that references it. I'd suggest adding the session
> cookie to your access log.
Given that we've definitely seen this happen with our sessionless application, I'm not sure that will help us much. For our other apps which have sessions, what happens if the incoming cookie references a session which has expired? Will the connection be simply handled by the new application? Is there some chance that the old application then creates a new valid session for that connection?
What we did do is add a version to our log4j logger so that we could see which application was handling requests. We got quite a mixture of requests to the older sessionless application.
>> 2. Is there some way to get the old application to fully undeploy as
>> soon as it has no live sessions?
>
> No. It is on the to do list.
Is there a task I can follow? I was unable to find one.
>> We have been thinking about writing an
>> external application to poll using JMX and do this, but that's quite a
>> bit of work. It would be nicer if this happened inside tomcat itself.
>
> Agreed. Patches welcome.
Sure. I understand, but I would not know where to begin with the tomcat codebase. I haven't even tried to read the source at this point. I assume we'd need to hook into some listener that detects when sessions are terminated and tell when they reach zero. That is maybe one line of code in the right place... or not :-)
>> 3. Is there some resource we might be hanging onto which is preventing
>> the old application from properly stopping?
>
> Maybe. But that would be a separate issue.
I am just looking for solutions within the particular stack that we are using (tapestry/cayenne) to see if there was some reason why tomcat wasn't fully letting go of the older application.
>> 4. Should the tomcat manager show the older apps as still running or
>> should they be shown as stopped?
>
> running.
It would be nice if it showed as "disabled" (to use the mod_jk terminology for an instance which is running but gets no new sessions). But that doesn't really affect our underlying problem: it would just make it easier to understand what is happening.
Thanks for your time.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: parallel deployment: multiple applications responding
Posted by Mark Thomas <ma...@apache.org>.
On 23/02/2012 06:51, Aristedes Maniatis wrote:
> We've been using parallel deployment for some time now with tomcat
> 7.0.25. For the most part the parallel deployment is a really nice idea,
> particularly because we don't have sessions serialised and clustered
> across all running instances.
>
> However, we've had mysterious problems for some time now where fixes
> we've applied to the application don't seem to work. We added some
> logging and have confirmed that some requests are still being served by
> the OLD instances of the application which are still deployed in the
> tomcat container but have an older version and therefore should not be
> responding. We might have two apps like this (using a format of YYMMDD
> and a two digit sequence):
>
> enrol##12022203.war
> enrol##12021503.war
>
> We would expect that requests would be served from only the newer
> application, but we find that requests continue to be served from the
> old, even though all the sessions to the old application are long dead.
>
> Just to confirm things we have another application which does not use
> sessions at all. It also has the same problem: requests are being served
> by the wrong application.
>
>
> In Tomcat Manager, the old application is still marked as running, but
> we thought this is just how it appears in the UI. Clearly there is
> something not right here.
>
>
> 1. Is this a known issue?
We'd need to understand the root cause to answer that. The short version
is that requests will only be routed to the old version if a request
includes a session ID that references it. I'd suggest adding the session
cookie to your access log.
> 2. Is there some way to get the old application to fully undeploy as
> soon as it has no live sessions?
No. It is on the to do list.
> We have been thinking about writing an
> external application to poll using JMX and do this, but that's quite a
> bit of work. It would be nicer if this happened inside tomcat itself.
Agreed. Patches welcome.
> 3. Is there some resource we might be hanging onto which is preventing
> the old application from properly stopping?
Maybe. But that would be a separate issue.
> 4. Should the tomcat manager show the older apps as still running or
> should they be shown as stopped?
running.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org