server-side endpoint cleanup problem in spring based CXF scenarios in OSGi

[Aki Yoshida <el...@googlemail.com>]

Hi,
There seems to be a problem in working with a spring based CXF
server-side endpoint in OSGi that can lead to a serious stability
issue. I would like to bring up this problem here for discussion so
that we can find a solution to it.

We can expect that an endpoint bundle (or a CXF scenario bundle,
whatever you call it) does not get automatically stopped when the cxf
bundle is stopped, if the scenario bundle is configured using spring
instead of blueprint. But whichever configuration is used, the
scenario bundle should be releasing all its resources when it is
stopped while the cxf bundle is already stopped.  However, it appears
to be that some resources are not released in this case. And I have
been thinking about a way to fix this issue. This problem is only
relevant when the cxf-bundle is just stopped without a
uninstall/install cycle.

Basically, what appears to be happening is, when the cxf bundle is
started, it instantiates a DestinationRegistryImpl instance and a
CXFNonSpringServlet servlet instance. And they are exposed as OSGi
services. When a scenario bundle is started, it instantiates an
AbstractHTTPDestination, registers it in the servlet’s destination
registry, and stores additional server-side bookkeeping information in
the CXFBusLifeCycleManager.

When the scenario bundle is stopped, the CXFBusLifeCycleManager
executes a series of cleanup actions which are grouped in the
preShutdown and postShutdown phases. In the preShutdown phase, the
corresponding server instance, ServerImpl, is stopped and this in turn
calls the destination’s (AbstractHTTPDestinations’) shutdown method.
This method accesses the registry object and removes the corresponding
path from it. This registry object is exposed as a proxy to the
scenario bundle and accessible while the cxf-bundle is active. When
the cxf-bundle is no longer active (e.g., stopped before), the thread
accessing this object simply gets blocked and never comes back alive.

Consequently, the clean up code placed in the subsequent phases is not
executed and this prevents some objects such as the bus and endpoint
instances to be released from memory. The destination registry is
probably also in some strange state as it goes stale before getting
emptied. The very bad side of this problem is that you don’t see any
errors in the log nor in the console. You only notice that your
console gets slow after a few start/stop cycles. You can observe the
manifestation of this problem when you use JMX or some analytical
tool.

As I mentioned earlier, I have thought about a way to prevent this
failed clean up step and tried the following approach that seems to
solve this issue.

In short, I added the appropriate cleanup code in the destroy method
of CXFNonSpringServlet and slightly modified AbstractHTTPDestination.
The long version follows.

First of all, the destination registry object is held in
CXFNonSpringServlet. I think this registry can get cleaned up when
this servlet itself is no longer in use (when the cxf-bundle is
stopped and the web container unregisters this servlet). Therefore, I
added in its destroy method the appropriate cleanup code for its
destination registry. And as I needed a way to mark the corresponding
destination (AbstractHTTPDestination) as “done/unregistered” so that
when the scenario bundle is stopped and the destination object is to
be cleaned up, the executing thread does not access the stale registry
reference attached to this destination object, I added method
releaseRegistry() in AbstractHTTPDestnation to set its registry to
null and changed the shutdown method of AbstractHTTPDestination to
accesses the registry reference and uses it only when the registry
reference is not null. With these changes, I am observing that the
scenario bundle can complete its clean up steps even when the
cxf-bundle is already stopped, thereby avoiding the problem described.

I am not sure if there is some problem with this approach or if there
are better approaches. I would appreciate your comments.

Thanks.
Regards, aki

Re: server-side endpoint cleanup problem in spring based CXF scenarios in OSGi

[Aki Yoshida <el...@googlemail.com>]

2011/7/6 Willem Jiang <wi...@gmail.com>:
> On 7/6/11 8:39 PM, Aki Yoshida wrote:
>>
>> Hi,
>> There seems to be a problem in working with a spring based CXF
>> server-side endpoint in OSGi that can lead to a serious stability
>> issue. I would like to bring up this problem here for discussion so
>> that we can find a solution to it.
>>
>> We can expect that an endpoint bundle (or a CXF scenario bundle,
>> whatever you call it) does not get automatically stopped when the cxf
>> bundle is stopped, if the scenario bundle is configured using spring
>> instead of blueprint. But whichever configuration is used, the
>> scenario bundle should be releasing all its resources when it is
>> stopped while the cxf bundle is already stopped.  However, it appears
>> to be that some resources are not released in this case. And I have
>> been thinking about a way to fix this issue. This problem is only
>> relevant when the cxf-bundle is just stopped without a
>> uninstall/install cycle.
>
> In most case we just test the CXF scenario bundle install and uninstall.
> So my question do you want the endpoint of CXF scenario bundle can be
> stopped when the CXF bundle is stopped ?

No, I am considering the special case when this type of automatic
start/stop dependency is not provided (i.e., not using blueprint nor
using a custom tracker to provide such automatic mechanism). Even in
this case, I think any potential event that might occur during the
operation of the system (i.e, administrative action or some failure)
should not lead to some unrecoverable inconsistency in the system that
requires the restart of the whole system. There may be some cases
where the restart of the whole system is inevitable. But we should at
least try to minimize such cases to happen.

>>
>> Basically, what appears to be happening is, when the cxf bundle is
>> started, it instantiates a DestinationRegistryImpl instance and a
>> CXFNonSpringServlet servlet instance. And they are exposed as OSGi
>> services. When a scenario bundle is started, it instantiates an
>> AbstractHTTPDestination, registers it in the servlet’s destination
>> registry, and stores additional server-side bookkeeping information in
>> the CXFBusLifeCycleManager.
>>
>> When the scenario bundle is stopped, the CXFBusLifeCycleManager
>> executes a series of cleanup actions which are grouped in the
>> preShutdown and postShutdown phases. In the preShutdown phase, the
>> corresponding server instance, ServerImpl, is stopped and this in turn
>> calls the destination’s (AbstractHTTPDestinations’) shutdown method.
>> This method accesses the registry object and removes the corresponding
>> path from it. This registry object is exposed as a proxy to the
>> scenario bundle and accessible while the cxf-bundle is active. When
>> the cxf-bundle is no longer active (e.g., stopped before), the thread
>> accessing this object simply gets blocked and never comes back alive.
>>
> That is not a good thing.
>
>> Consequently, the clean up code placed in the subsequent phases is not
>> executed and this prevents some objects such as the bus and endpoint
>> instances to be released from memory. The destination registry is
>> probably also in some strange state as it goes stale before getting
>> emptied. The very bad side of this problem is that you don’t see any
>> errors in the log nor in the console. You only notice that your
>> console gets slow after a few start/stop cycles. You can observe the
>> manifestation of this problem when you use JMX or some analytical
>> tool.
>>
>> As I mentioned earlier, I have thought about a way to prevent this
>> failed clean up step and tried the following approach that seems to
>> solve this issue.
>>
>> In short, I added the appropriate cleanup code in the destroy method
>> of CXFNonSpringServlet and slightly modified AbstractHTTPDestination.
>> The long version follows.
>>
>> First of all, the destination registry object is held in
>> CXFNonSpringServlet. I think this registry can get cleaned up when
>> this servlet itself is no longer in use (when the cxf-bundle is
>> stopped and the web container unregisters this servlet). Therefore, I
>> added in its destroy method the appropriate cleanup code for its
>> destination registry. And as I needed a way to mark the corresponding
>> destination (AbstractHTTPDestination) as “done/unregistered” so that
>> when the scenario bundle is stopped and the destination object is to
>> be cleaned up, the executing thread does not access the stale registry
>> reference attached to this destination object, I added method
>> releaseRegistry() in AbstractHTTPDestnation to set its registry to
>> null and changed the shutdown method of AbstractHTTPDestination to
>> accesses the registry reference and uses it only when the registry
>> reference is not null. With these changes, I am observing that the
>> scenario bundle can complete its clean up steps even when the
>> cxf-bundle is already stopped, thereby avoiding the problem described.
>>
> It sounds good.
> I just checked the code of JettyHTTPDestination, it will not be affect by
> this code, if CXF bundle already clean up the Registry.
>
>> I am not sure if there is some problem with this approach or if there
>> are better approaches. I would appreciate your comments.
>>
> I just have a timing issue, what if the CXF scenario bundle is stopped just
> after the CXF bundle in the other thread.

This is a good point. I am not sure when exactly this registry
reference goes stale during the stopping process. There may be some
small timing gap in which the scenario bundle could access the
registry reference at the beginning of its cleanup process but
somewhere in the middle of executing its method, as the cxf bundle is
stopped by a separate thread, it may get stuck. I don't know what the
compliant implementation is supposed to behave in such cases. But I
can experiment with this situation to see what can happen.

regards, aki

>
>
> --
> Willem
> ----------------------------------
> FuseSource
> Web: http://www.fusesource.com
> Blog:    http://willemjiang.blogspot.com (English)
>         http://jnn.javaeye.com (Chinese)
> Twitter: willemjiang
> Weibo: willemjiang
>

Re: server-side endpoint cleanup problem in spring based CXF scenarios in OSGi

[Willem Jiang <wi...@gmail.com>]

On 7/6/11 8:39 PM, Aki Yoshida wrote:
> Hi,
> There seems to be a problem in working with a spring based CXF
> server-side endpoint in OSGi that can lead to a serious stability
> issue. I would like to bring up this problem here for discussion so
> that we can find a solution to it.
>
> We can expect that an endpoint bundle (or a CXF scenario bundle,
> whatever you call it) does not get automatically stopped when the cxf
> bundle is stopped, if the scenario bundle is configured using spring
> instead of blueprint. But whichever configuration is used, the
> scenario bundle should be releasing all its resources when it is
> stopped while the cxf bundle is already stopped.  However, it appears
> to be that some resources are not released in this case. And I have
> been thinking about a way to fix this issue. This problem is only
> relevant when the cxf-bundle is just stopped without a
> uninstall/install cycle.
In most case we just test the CXF scenario bundle install and uninstall.
So my question do you want the endpoint of CXF scenario bundle can be 
stopped when the CXF bundle is stopped ?

>
> Basically, what appears to be happening is, when the cxf bundle is
> started, it instantiates a DestinationRegistryImpl instance and a
> CXFNonSpringServlet servlet instance. And they are exposed as OSGi
> services. When a scenario bundle is started, it instantiates an
> AbstractHTTPDestination, registers it in the servlet’s destination
> registry, and stores additional server-side bookkeeping information in
> the CXFBusLifeCycleManager.
>
> When the scenario bundle is stopped, the CXFBusLifeCycleManager
> executes a series of cleanup actions which are grouped in the
> preShutdown and postShutdown phases. In the preShutdown phase, the
> corresponding server instance, ServerImpl, is stopped and this in turn
> calls the destination’s (AbstractHTTPDestinations’) shutdown method.
> This method accesses the registry object and removes the corresponding
> path from it. This registry object is exposed as a proxy to the
> scenario bundle and accessible while the cxf-bundle is active. When
> the cxf-bundle is no longer active (e.g., stopped before), the thread
> accessing this object simply gets blocked and never comes back alive.
>
That is not a good thing.

> Consequently, the clean up code placed in the subsequent phases is not
> executed and this prevents some objects such as the bus and endpoint
> instances to be released from memory. The destination registry is
> probably also in some strange state as it goes stale before getting
> emptied. The very bad side of this problem is that you don’t see any
> errors in the log nor in the console. You only notice that your
> console gets slow after a few start/stop cycles. You can observe the
> manifestation of this problem when you use JMX or some analytical
> tool.
>
> As I mentioned earlier, I have thought about a way to prevent this
> failed clean up step and tried the following approach that seems to
> solve this issue.
>
> In short, I added the appropriate cleanup code in the destroy method
> of CXFNonSpringServlet and slightly modified AbstractHTTPDestination.
> The long version follows.
>
> First of all, the destination registry object is held in
> CXFNonSpringServlet. I think this registry can get cleaned up when
> this servlet itself is no longer in use (when the cxf-bundle is
> stopped and the web container unregisters this servlet). Therefore, I
> added in its destroy method the appropriate cleanup code for its
> destination registry. And as I needed a way to mark the corresponding
> destination (AbstractHTTPDestination) as “done/unregistered” so that
> when the scenario bundle is stopped and the destination object is to
> be cleaned up, the executing thread does not access the stale registry
> reference attached to this destination object, I added method
> releaseRegistry() in AbstractHTTPDestnation to set its registry to
> null and changed the shutdown method of AbstractHTTPDestination to
> accesses the registry reference and uses it only when the registry
> reference is not null. With these changes, I am observing that the
> scenario bundle can complete its clean up steps even when the
> cxf-bundle is already stopped, thereby avoiding the problem described.
>
It sounds good.
I just checked the code of JettyHTTPDestination, it will not be affect 
by this code, if CXF bundle already clean up the Registry.

> I am not sure if there is some problem with this approach or if there
> are better approaches. I would appreciate your comments.
>
I just have a timing issue, what if the CXF scenario bundle is stopped 
just after the CXF bundle in the other thread.

> Thanks.
> Regards, aki
>


-- 
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog:    http://willemjiang.blogspot.com (English)
          http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang