You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aries.apache.org by Christian Schneider <ch...@die-schneider.net> on 2015/02/10 09:15:54 UTC
Do we really need Quiesce support?
In several places in aries we support the Quiesce API. As far as I
understood its purpose is to shut down bundles without actually stopping
them. It does not seem to be any standard as the interfaces are in the
Aries namespace.
The problem with it is that it complicates the code a lot. For example
in jpa container we create a proxy for each EntityManager that also
supports quiesce methods even though they are
not part of the EntityManager interface. The code in other modules is
similarly affected.
Besides this I do not see anyone using the API. I am also not sure why
we need it at all as OSGi already has nice lifecycle methods to stop
bundles which should have a similar effect if implemented nicely. One
other disadvantge of Quiesce is that you can only shut down a bundle.
You can not restart it again.
So the question is: Do we really need to support the Quiesce API? The
code would be a lot less complex and error prone without it.
WDYT?
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Christian,
I don't think we use the Quiesce API (at least in Karaf). Maybe some
other users use it, let's see ;)
Regards
JB
On 02/10/2015 09:15 AM, Christian Schneider wrote:
> In several places in aries we support the Quiesce API. As far as I
> understood its purpose is to shut down bundles without actually stopping
> them. It does not seem to be any standard as the interfaces are in the
> Aries namespace.
>
> The problem with it is that it complicates the code a lot. For example
> in jpa container we create a proxy for each EntityManager that also
> supports quiesce methods even though they are
> not part of the EntityManager interface. The code in other modules is
> similarly affected.
>
> Besides this I do not see anyone using the API. I am also not sure why
> we need it at all as OSGi already has nice lifecycle methods to stop
> bundles which should have a similar effect if implemented nicely. One
> other disadvantge of Quiesce is that you can only shut down a bundle.
> You can not restart it again.
>
> So the question is: Do we really need to support the Quiesce API? The
> code would be a lot less complex and error prone without it.
>
> WDYT?
>
> Christian
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-13 22:56 GMT+01:00 David Bosschaert <da...@gmail.com>:
> I think I understand the use-case now as well. The osgi-dev thread
> started by Christian [1] also provides valuable insight.
> If I can summarize the quiesce functionality it is aimed at preventing
> operations from failing. Rather than failing, you stop accepting
> requests, then perform some interruptive task such as applying an
> update and then you continue.
>
Agreed
>
> I'm wondering however, while this might prevent a certain form of
> failure, other forms of failure might still exist. Someone might
> accidentally unplug a network cable. A harddisk (or even raid array)
> crashes. There is a power outage. It's quite easy to come up with
> other examples. Even with quiesce support failure might still happen
> as the bundle doesn't accept requests any more and hence the caller
> might fail.
>
I don't think we should even think about using the quiesce api
in such a case. The quiesce api is really to clean stop a bunch
of bundles. This means that a user, or agent, decided to stop it.
In the case of an outage, the system must react to an error condition,
but this is usually already managed in most cases.
For example a JDBC connection will thrown an exception, and when
the application try to use it again, it will have to wait to establish a
new connection.
So I don't think we should even think about this problem, that's out
of the scope.
I can summarise the requirements:
R1: we want a clean shutdown where inflight calls are correctly finished
R2: stopped bundles should not perform any more work, as they should
have clean up all their resources by the time they are stopped
R3: we have a bigger problem when we consider a set of bundles to be
stopped as an inflight call in one bundle may lead to new calls
in
some service dependencies
R4: we also need a way to have a fast shutdown
R4 could be removed, in which case, the logic to cleanly stop a
bundle could be done when the bundle is actually stopped (i.e. in the
BundleActivator#stop in case of pure osgi bundle or by intercepting the
STOPPING event in the case of an extender).
If R4 is kept, it means we need something like the QuiesceParticipant
that bundles or extenders could implement to perform a clean shutdown.
Another alternative would be to always do the logic when the bundle is
stopped, and using some kind of global flag. The downside is that if you
use the same method, you loose the ability to perform a quick stop if
a clean stop is already running (the clean stop could take quite some time).
R3 means we need an API (it could be an OSGi service as in the quiesce
api, or some reusable code).
Given we're mostly talking about service dependencies, I think the code I
wrote in karaf4 could be really reused to order bundles correctly. I think
the current quiesce manager does not cover this use case correctly, but I
may be wrong.
Also, we need to consider other extenders such as servlets, and DS.
For blueprint, we already have some code and it's easily doable because
blueprint already has support for proxies. DS does not have that and the
only way with the current code would rely on the user to add its own
synchronization to delay the deactivation of each component until calls are
finished.
Anyway, the current quiesce API is quite minimal and seems to fulfil the
problem to me. The code may be improved obviously and we could
enhance other extenders to actually leverage it (felix scr, pax-web,
etc...).
>
> As Tim Diekmann suggested on the osgi-dev thread, to really be
> completely safe you need to build your safety on another level. It
> might be better to write your system such that it is ready for
> (occasional) failure and can handle that gracefully and correctly
> rather than trying to come up with solutions that might prevent some
> of those occasional failures, especially since it can never handle all
> of them.
>
As I said earlier, I don't see any relationship between performing a clean
shut down of bundles and reacting to any kind of system failure. The
clean shut down is to be used when there's no outage and imho it does
not make any sense at all to use it in case of an outage.
>
> Just my 2c,
>
> David
>
> [1] https://mail.osgi.org/pipermail/osgi-dev/2015-February/004744.html
>
> On 13 February 2015 at 18:25, Christian Schneider
> <ch...@die-schneider.net> wrote:
> > Many thanks for the explanations.
> >
> > For JPA I think I now understand what we need to achieve. As soon as a
> > persistence unit bundle is quiesced no new EMs should be created.
> > The opened EMs of the persistence bundle are closed when all calls that
> use
> > them are finished. Then when all those EMs are closed the quiesce
> > operation reports that it is finished.
> >
> > I hope this is stated correctly.
> >
> > Btw. the reason why I want to really understand this in detail is that I
> am
> > planning a bigger redesign of the jpa code. So I want to make sure I
> > understand the current requirements and
> > also try to slim them down wherever possible.
> >
> > Christian
> >
> >
> > On 13.02.2015 17:56, Jeremy Hughes wrote:
> >>
> >> I've been digging around in my memory. The Quiesce capability is a like
> a
> >> soft-stop. It's a generic capability that is enforced differently
> >> depending
> >> on the context. It's more normal for an extender to implement the
> >> QuiesceParticipant rather than an individual bundle. In the case of the
> >> Blueprint extender, in flight service calls are allowed to finish -
> >> they're
> >> counted in and counted out - and no new calls are allowed. When there
> are
> >> no in flight service calls, the bundle is told to stop. The Blueprint
> >> bundle itself doesn't know this is happening which is a key design
> point.
> >> In the case of JPA, I believe the extender ensures no new Entity
> Managers
> >> are created - so similar to the Blueprint case. But it's been a while
> >> since
> >> I looked at this.
> >>
> >> Hope that helps.
> >> Jeremy
> >>
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
>
Re: Do we really need Quiesce support?
Posted by David Bosschaert <da...@gmail.com>.
I think I understand the use-case now as well. The osgi-dev thread
started by Christian [1] also provides valuable insight.
If I can summarize the quiesce functionality it is aimed at preventing
operations from failing. Rather than failing, you stop accepting
requests, then perform some interruptive task such as applying an
update and then you continue.
I'm wondering however, while this might prevent a certain form of
failure, other forms of failure might still exist. Someone might
accidentally unplug a network cable. A harddisk (or even raid array)
crashes. There is a power outage. It's quite easy to come up with
other examples. Even with quiesce support failure might still happen
as the bundle doesn't accept requests any more and hence the caller
might fail.
As Tim Diekmann suggested on the osgi-dev thread, to really be
completely safe you need to build your safety on another level. It
might be better to write your system such that it is ready for
(occasional) failure and can handle that gracefully and correctly
rather than trying to come up with solutions that might prevent some
of those occasional failures, especially since it can never handle all
of them.
Just my 2c,
David
[1] https://mail.osgi.org/pipermail/osgi-dev/2015-February/004744.html
On 13 February 2015 at 18:25, Christian Schneider
<ch...@die-schneider.net> wrote:
> Many thanks for the explanations.
>
> For JPA I think I now understand what we need to achieve. As soon as a
> persistence unit bundle is quiesced no new EMs should be created.
> The opened EMs of the persistence bundle are closed when all calls that use
> them are finished. Then when all those EMs are closed the quiesce
> operation reports that it is finished.
>
> I hope this is stated correctly.
>
> Btw. the reason why I want to really understand this in detail is that I am
> planning a bigger redesign of the jpa code. So I want to make sure I
> understand the current requirements and
> also try to slim them down wherever possible.
>
> Christian
>
>
> On 13.02.2015 17:56, Jeremy Hughes wrote:
>>
>> I've been digging around in my memory. The Quiesce capability is a like a
>> soft-stop. It's a generic capability that is enforced differently
>> depending
>> on the context. It's more normal for an extender to implement the
>> QuiesceParticipant rather than an individual bundle. In the case of the
>> Blueprint extender, in flight service calls are allowed to finish -
>> they're
>> counted in and counted out - and no new calls are allowed. When there are
>> no in flight service calls, the bundle is told to stop. The Blueprint
>> bundle itself doesn't know this is happening which is a key design point.
>> In the case of JPA, I believe the extender ensures no new Entity Managers
>> are created - so similar to the Blueprint case. But it's been a while
>> since
>> I looked at this.
>>
>> Hope that helps.
>> Jeremy
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
Many thanks for the explanations.
For JPA I think I now understand what we need to achieve. As soon as a
persistence unit bundle is quiesced no new EMs should be created.
The opened EMs of the persistence bundle are closed when all calls that
use them are finished. Then when all those EMs are closed the quiesce
operation reports that it is finished.
I hope this is stated correctly.
Btw. the reason why I want to really understand this in detail is that I
am planning a bigger redesign of the jpa code. So I want to make sure I
understand the current requirements and
also try to slim them down wherever possible.
Christian
On 13.02.2015 17:56, Jeremy Hughes wrote:
> I've been digging around in my memory. The Quiesce capability is a like a
> soft-stop. It's a generic capability that is enforced differently depending
> on the context. It's more normal for an extender to implement the
> QuiesceParticipant rather than an individual bundle. In the case of the
> Blueprint extender, in flight service calls are allowed to finish - they're
> counted in and counted out - and no new calls are allowed. When there are
> no in flight service calls, the bundle is told to stop. The Blueprint
> bundle itself doesn't know this is happening which is a key design point.
> In the case of JPA, I believe the extender ensures no new Entity Managers
> are created - so similar to the Blueprint case. But it's been a while since
> I looked at this.
>
> Hope that helps.
> Jeremy
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Jeremy Hughes <jp...@gmail.com>.
I've been digging around in my memory. The Quiesce capability is a like a
soft-stop. It's a generic capability that is enforced differently depending
on the context. It's more normal for an extender to implement the
QuiesceParticipant rather than an individual bundle. In the case of the
Blueprint extender, in flight service calls are allowed to finish - they're
counted in and counted out - and no new calls are allowed. When there are
no in flight service calls, the bundle is told to stop. The Blueprint
bundle itself doesn't know this is happening which is a key design point.
In the case of JPA, I believe the extender ensures no new Entity Managers
are created - so similar to the Blueprint case. But it's been a while since
I looked at this.
Hope that helps.
Jeremy
On 13 February 2015 at 11:43, Guillaume Nodet <gn...@apache.org> wrote:
> 2015-02-13 12:23 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
>
> > On 13.02.2015 11:58, Guillaume Nodet wrote:
> >
> >>
> >>
> >> Your example expects a simple restart of the service, I'm talking about
> an
> >> update/refresh of the bundle.
> >> But even restarting the service, if the service is an EntityManager,
> that
> >> does not work afaik : you can't just switch it in the middle of your
> >> processing (what you've already done with the JDBC connection in the
> >> context of your transaction can't be propagated to a new connection).
> >>
> > You are right about a service that provides EntityManger instances. The
> > problem here is that the service is not really stateless which makes it
> > much harder to manage.
> >
>
> You could also think about a camel component you want to update. In such a
> case, you want the routes to cleanly shutdown before actually updating the
> component.
>
>
> > For a normal EntityManagerFactory (like provided by jpa.container) this
> > should not apply as the EntityManager is under control of the client
> bundle.
> >
> > In case of jpa.container.context there probably is a problem. The service
> > holds instances of EntityManager. So if this bundle is updated I would
> > expect it to close
> > all EntityManager instances. So the state would be lost. This is a very
> > special case though and I dont think it would warant the Quiesce support
> > all for itself.
> > After all updating aries jpa is rather seldom for a customer compared to
> > updating their own bundles. So I think it should be ok to handle this
> case
> > manually in a special way.
> >
> > For updates of user bundles there should be no issue I think.
> >
>
> This problem is a general problem, think about CXF services, camel routes,
> etc...
>
> It may be possible to partially get rid of the quiesce api if all bundles
> would always perform a clean shutdown when being stopped. That's actually
> what camel does afaik.
> If shutting down a blueprint bundle or an jpa container would wait for all
> calls to be finished before actually shutting down, it may help a lot.
> Though even doing that would not really be sufficient, as you can't track
> inside an individual bundle, the fact that a service is "still" in use,
> even without being currently called.
> I.e. if you have a service which performs several operations on the entity
> manager, you could delay the stop of the entity manager until one call is
> finished, but what you really want is coordination so that the entity
> manager it stopped until the bundle using the entity manager is completely
> done. For that, you need cooperation of bundles, and the OSGi api does not
> really provide anything afaik.
>
> There may be alternatives though, as the service dependencies can be
> introspected from the outside.
> In karaf 4, when we need to refresh bundles, we do an orderly shut down of
> bundles according to service dependencies. In the case above, we would
> first shut down the client bundle, then the JPA entity manager. If each
> bundle performs a clean shut down, it may be sufficient and we could get
> rid of the quiesce api, but not of the code which tracks calls, that's
> needed to perform the clean shut down.
>
>
> >
> > Btw. I am not yet sure what a refresh in OSGi actually does. Let us
> assume
> > we refresh a bundle that provides a package that is imported by other
> > bundles. Will these bundles also be refreshed then?
> >
> Yes
>
> > If not then I think these bundles will not see the new package. So in
> > case the package contains a service interface they will not see the new
> > service. As this seems to work in practice (at least in karaf) I assume
> > that in karaf the
> > client bundles are also refreshed. Is that correct?
> >
>
> If the interface containing the bundle is refreshed, the bundle using that
> interface is refreshed too. If the api is packaged as a separate bundle
> and you update the implementation, the client bundle won't be refreshed.
>
> >
> > Btw. this reminds me of this issue https://issues.apache.org/
> > jira/browse/ARIES-1270 .
> > There we refresh a persistence unit bundle. After this the client bundle
> > and the EntityManagerFactory saw different classes for the Entities.
> > I do not think this is Quiesce related though. If you have any ideas
> about
> > that would be great as I do not seem to find any solution.
> >
> > The doc mentions "threads started by this bundle".
> >>>
> >>> If a bundle B calls a service from bundle A then the thread is not
> >>> started
> >>> by bundle A. So it should be fine that this thread keeps running
> >>> even when the Activator.stop was called.
> >>>
> >>> Here's what the spec says for services (section 5.7, osgi core 6):
> the
> >> behaviour of a a service object that becomes unregistered is undefined.
> >> Such service objects may continue to work properly or throw an exception
> >> at
> >> their discretion. Imho, in a production environment, you don't really
> >> want
> >> to rely on an undefined behavior. And if the service throws an
> exception,
> >> it's not a clean shutdown anymore.
> >>
> > I would interpret this part of the spec ("at their discretion") that the
> > service implementor is free to either continue working or throw an
> > exception.
> > So if he does not do anything special it should be fine.
> >
> >
> > Christian
> >
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
> >
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
On 13.02.2015 17:40, David Jencks wrote:
> I don't know what the quiesce service is supposed to do either. However I take issue with one statement below… hopefully I misunderstood what you mean:
>> - bundleA stops which should not affect inflight processing inside bundle A
>
> ????????????
>
> Lets suppose A's services are implemented as DS components. Stopping the bundle will deactivate and unregister all the services. Unless someone has taken care (probably with a read-write lock on all the DS event and lifecycle methods and all the service operations, something I have never ever seen, and would violate the "quick shutdown" expectation) that all the work being done on worker threads has completed before deactivate is called, worker threads will continue to operate in the deactivate component. They aren't going to get very far because all the references have been removed.
>
> So I rather suspect the quiesce service might be intended to deal with this kind of problem, although I have no idea how.
For DS I am not sure what it does when it shuts down a component. If it
simply deregisters the component then its injected service objects would
still be present. So the inflight call might still work. I am not sure
about this though.
For blueprint I am also not sure if the inject proxies for services
would still work. Probably not.
So yes you are right. In the general case inflight calls might indeed be
disturbed. In this case I wonder how this is expected to work in OSGi
without an additional API.
I also just discussed with Guillaume on IRC that at least after
uninstall even the classloader of the bundle might stop working.
I asked about these cases in the osgi dev list to see if plain OSGi has
some answers for these problems.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by David Jencks <da...@yahoo.com.INVALID>.
I don't know what the quiesce service is supposed to do either. However I take issue with one statement below… hopefully I misunderstood what you mean:
> - bundleA stops which should not affect inflight processing inside bundle A
????????????
Lets suppose A's services are implemented as DS components. Stopping the bundle will deactivate and unregister all the services. Unless someone has taken care (probably with a read-write lock on all the DS event and lifecycle methods and all the service operations, something I have never ever seen, and would violate the "quick shutdown" expectation) that all the work being done on worker threads has completed before deactivate is called, worker threads will continue to operate in the deactivate component. They aren't going to get very far because all the references have been removed.
So I rather suspect the quiesce service might be intended to deal with this kind of problem, although I have no idea how.
thanks
david jencks
On Feb 13, 2015, at 5:17 AM, Christian Schneider <ch...@die-schneider.net> wrote:
> On 13.02.2015 10:43, Guillaume Nodet wrote:
>>
>>
>> I think you did not understand the concept of *clean* shutdown which I'm
>> trying to explain.
>> I think there's a big difference between reacting to the fact that the
>> database connection has been lost, which is out of your control and the
>> only thing you can do is stop your own services asap, and cleanly shut down
>> your application.
>>
>> Let's tale again my example where we have bundle B which is a service
>> exposed to the outside world and bundle A, a JPA layer. If I want to
>> update bundle A and I simply call bundleA.update(), this will cause users
>> of bundleB to have their calls fail. What we really need is the following:
>> * stop any new users to use bundle B
>> * wait until all calls from bundle B to finish
>> * stop bundles A and B (which are not actively used anymore)
>> * update bundle A
>>
> I think an update of bundleA should simply work without issues.
>
> From what I understood the following would happen.
> - bundleA stops which should not affect inflight processing inside bundle A
> - If bundle B uses blueprint then new calls to bundle A will block
> - bundleA is replaced and started
> - Calls from bundle B continue again
>
> So I think we do not have a problem in this case.
>
>
>>>
>>> This means that in order to stop A, I think you want to quiesce A and B.
>>>> One problem with blueprint, and it may be an implementation problem, is the
>>>> following. I just made a test with the blueprint-testquiescebundle which
>>>> exposes a simple bean with a sleep method through blueprint. If I call it
>>>> with a long sleep, and I concurrently stop the bundle, the bean will
>>>> continue executing while the bundle is already stopped. That's quite bad
>>>> actually.
>>>>
>>> I think it is absolutely fine that threads of a bundle that stops or is
>>> even uninstalled continue to run. We might want to interrupt them so a
>>> sleep returns earlier but even if the sleep continues I do not see an
>>> immediate problem. It would just prevent the classloader from cleaning up
>>> the bundle classses for some time. Btw. I think having such a long sleep is
>>> an implementation problem that OSGi does not need to fix.
>>
>> Uh, no that's clearly not fine. BundleActivator#stop() says the following:
>>
>> /**
>> * Called when this bundle is stopped so the Framework can perform the
>> * bundle-specific activities necessary to stop the bundle. In general, this
>> * method should undo the work that the {@code BundleActivator.start}
>> * method started. There should be no active threads that were started by
>> * this bundle when this bundle returns. A stopped bundle must not call any
>> * Framework objects.
>> *
>> * <p>
>> * This method must complete and return to its caller in a timely manner.
>> */
>>
>> So, we can infer the following things:
>> * you're not allowed to wait for quite a long time to cleanly wait for
>> things to be ready to be stopped
>> * you're not allowed to leave threads running
>>
>> This leads to the conclusion that the OSGi api is not sufficient to cover
>> clean shutdown of bundles. I think the quiesce api aims to solve this
>> problem. I'm not saying it can't be improved or implemented differently,
>> I'm just trying to expose what I think is the purpose of this api.
> The doc mentions "threads started by this bundle".
>
> If a bundle B calls a service from bundle A then the thread is not started by bundle A. So it should be fine that this thread keeps running
> even when the Activator.stop was called.
>
> So I think the OSGi API is fine and I still see no need for Quiesce.
> ... and honestly if the OSGi was not able to cover this quite typical case then it would be really bad and we should not use OSGi.
> If we are not sure then I propose we forward this question to the OSGi dev list and ask for some advice there.
>
>>
>>
>>
>> I agree, I'm not sure to fully understand it. That does not mean we should
>> get rid of it just because we don't understand it ;-)
> I agree. The problem is though if we have no one who understands it then we have a big problem when moving on.
>
> So my point of view is that we either have someone who understands it or at least have the use cases documented. If both is not the case then
> I indeed would rather like to get rid of Quiesce. Of course not in a bug fix or minor version. It should be ok for a major version though.
>
> Christian
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
On 13.02.2015 12:43, Guillaume Nodet wrote:
> 2015-02-13 12:23 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
>
>> On 13.02.2015 11:58, Guillaume Nodet wrote:
>>
>>>
>>> Your example expects a simple restart of the service, I'm talking about an
>>> update/refresh of the bundle.
>>> But even restarting the service, if the service is an EntityManager, that
>>> does not work afaik : you can't just switch it in the middle of your
>>> processing (what you've already done with the JDBC connection in the
>>> context of your transaction can't be propagated to a new connection).
>>>
>> You are right about a service that provides EntityManger instances. The
>> problem here is that the service is not really stateless which makes it
>> much harder to manage.
>>
> You could also think about a camel component you want to update. In such a
> case, you want the routes to cleanly shutdown before actually updating the
> component.
For camel I am not sure. The components are looked up as services but
camel does not really cope
with OSGi dynamics afaik. So indeed updating a component might very well
lead to problems.
Still this would be no clear reason for using Quiesce for me. The camel
extender would see that the component is gone and could
stop/restart all bundles that need the component.
I would be careful to put too much logic into this though. As the
problem is probably not to common.
>> For updates of user bundles there should be no issue I think.
>>
> This problem is a general problem, think about CXF services, camel routes,
> etc...
>
> It may be possible to partially get rid of the quiesce api if all bundles
> would always perform a clean shutdown when being stopped. That's actually
> what camel does afaik.
> If shutting down a blueprint bundle or an jpa container would wait for all
> calls to be finished before actually shutting down, it may help a lot.
> Though even doing that would not really be sufficient, as you can't track
> inside an individual bundle, the fact that a service is "still" in use,
> even without being currently called.
> I.e. if you have a service which performs several operations on the entity
> manager, you could delay the stop of the entity manager until one call is
> finished, but what you really want is coordination so that the entity
> manager it stopped until the bundle using the entity manager is completely
> done. For that, you need cooperation of bundles, and the OSGi api does not
> really provide anything afaik.
If we are talking about an update of the user jpa client bundle then I
dont think the EntityManager is
an issue. The user jpa client bundle does not control the EntityManager.
So updating the user bundle should even
keep the EntityManager intact.
If the persistence unit bundle is updated then the whole
EntityManagerFactory has to be taken down of course.
All instances of EntityManager created by the old EMF should then also
be cleaned up.
This is indeed a bit tricky but I think even this case should work
without Quiesce. The critical part is that we may not close and
EntityManager instance
while a bundle is still using it inside a method.
So we need to know when a method that can use an EntityManager is
entered and when it is left. As we have an interceptor for these methods
we at least have the chance to do this.
So in the preCall we could count +1 for the EM of this thread and in
postCall we can count -1. An EntityManager then can only be closed when
the count is at 0.
I think this should provide us with enough information to cleanly shut
down EMs.
I am not sure about all the details but I think it should be doable.
I will try to implement this in the new design I experiment with.
>
> There may be alternatives though, as the service dependencies can be
> introspected from the outside.
> In karaf 4, when we need to refresh bundles, we do an orderly shut down of
> bundles according to service dependencies. In the case above, we would
> first shut down the client bundle, then the JPA entity manager. If each
> bundle performs a clean shut down, it may be sufficient and we could get
> rid of the quiesce api, but not of the code which tracks calls, that's
> needed to perform the clean shut down.
I am not sure about the general case but what you describe about karaf 4
already sounds quite solid.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-13 12:23 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
> On 13.02.2015 11:58, Guillaume Nodet wrote:
>
>>
>>
>> Your example expects a simple restart of the service, I'm talking about an
>> update/refresh of the bundle.
>> But even restarting the service, if the service is an EntityManager, that
>> does not work afaik : you can't just switch it in the middle of your
>> processing (what you've already done with the JDBC connection in the
>> context of your transaction can't be propagated to a new connection).
>>
> You are right about a service that provides EntityManger instances. The
> problem here is that the service is not really stateless which makes it
> much harder to manage.
>
You could also think about a camel component you want to update. In such a
case, you want the routes to cleanly shutdown before actually updating the
component.
> For a normal EntityManagerFactory (like provided by jpa.container) this
> should not apply as the EntityManager is under control of the client bundle.
>
> In case of jpa.container.context there probably is a problem. The service
> holds instances of EntityManager. So if this bundle is updated I would
> expect it to close
> all EntityManager instances. So the state would be lost. This is a very
> special case though and I dont think it would warant the Quiesce support
> all for itself.
> After all updating aries jpa is rather seldom for a customer compared to
> updating their own bundles. So I think it should be ok to handle this case
> manually in a special way.
>
> For updates of user bundles there should be no issue I think.
>
This problem is a general problem, think about CXF services, camel routes,
etc...
It may be possible to partially get rid of the quiesce api if all bundles
would always perform a clean shutdown when being stopped. That's actually
what camel does afaik.
If shutting down a blueprint bundle or an jpa container would wait for all
calls to be finished before actually shutting down, it may help a lot.
Though even doing that would not really be sufficient, as you can't track
inside an individual bundle, the fact that a service is "still" in use,
even without being currently called.
I.e. if you have a service which performs several operations on the entity
manager, you could delay the stop of the entity manager until one call is
finished, but what you really want is coordination so that the entity
manager it stopped until the bundle using the entity manager is completely
done. For that, you need cooperation of bundles, and the OSGi api does not
really provide anything afaik.
There may be alternatives though, as the service dependencies can be
introspected from the outside.
In karaf 4, when we need to refresh bundles, we do an orderly shut down of
bundles according to service dependencies. In the case above, we would
first shut down the client bundle, then the JPA entity manager. If each
bundle performs a clean shut down, it may be sufficient and we could get
rid of the quiesce api, but not of the code which tracks calls, that's
needed to perform the clean shut down.
>
> Btw. I am not yet sure what a refresh in OSGi actually does. Let us assume
> we refresh a bundle that provides a package that is imported by other
> bundles. Will these bundles also be refreshed then?
>
Yes
> If not then I think these bundles will not see the new package. So in
> case the package contains a service interface they will not see the new
> service. As this seems to work in practice (at least in karaf) I assume
> that in karaf the
> client bundles are also refreshed. Is that correct?
>
If the interface containing the bundle is refreshed, the bundle using that
interface is refreshed too. If the api is packaged as a separate bundle
and you update the implementation, the client bundle won't be refreshed.
>
> Btw. this reminds me of this issue https://issues.apache.org/
> jira/browse/ARIES-1270 .
> There we refresh a persistence unit bundle. After this the client bundle
> and the EntityManagerFactory saw different classes for the Entities.
> I do not think this is Quiesce related though. If you have any ideas about
> that would be great as I do not seem to find any solution.
>
> The doc mentions "threads started by this bundle".
>>>
>>> If a bundle B calls a service from bundle A then the thread is not
>>> started
>>> by bundle A. So it should be fine that this thread keeps running
>>> even when the Activator.stop was called.
>>>
>>> Here's what the spec says for services (section 5.7, osgi core 6): the
>> behaviour of a a service object that becomes unregistered is undefined.
>> Such service objects may continue to work properly or throw an exception
>> at
>> their discretion. Imho, in a production environment, you don't really
>> want
>> to rely on an undefined behavior. And if the service throws an exception,
>> it's not a clean shutdown anymore.
>>
> I would interpret this part of the spec ("at their discretion") that the
> service implementor is free to either continue working or throw an
> exception.
> So if he does not do anything special it should be fine.
>
>
> Christian
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
On 13.02.2015 11:58, Guillaume Nodet wrote:
>
>
> Your example expects a simple restart of the service, I'm talking about an
> update/refresh of the bundle.
> But even restarting the service, if the service is an EntityManager, that
> does not work afaik : you can't just switch it in the middle of your
> processing (what you've already done with the JDBC connection in the
> context of your transaction can't be propagated to a new connection).
You are right about a service that provides EntityManger instances. The
problem here is that the service is not really stateless which makes it
much harder to manage.
For a normal EntityManagerFactory (like provided by jpa.container) this
should not apply as the EntityManager is under control of the client bundle.
In case of jpa.container.context there probably is a problem. The
service holds instances of EntityManager. So if this bundle is updated I
would expect it to close
all EntityManager instances. So the state would be lost. This is a very
special case though and I dont think it would warant the Quiesce support
all for itself.
After all updating aries jpa is rather seldom for a customer compared to
updating their own bundles. So I think it should be ok to handle this
case manually in a special way.
For updates of user bundles there should be no issue I think.
Btw. I am not yet sure what a refresh in OSGi actually does. Let us
assume we refresh a bundle that provides a package that is imported by
other bundles. Will these bundles also be refreshed then?
If not then I think these bundles will not see the new package. So in
case the package contains a service interface they will not see the new
service. As this seems to work in practice (at least in karaf) I assume
that in karaf the
client bundles are also refreshed. Is that correct?
Btw. this reminds me of this issue
https://issues.apache.org/jira/browse/ARIES-1270 .
There we refresh a persistence unit bundle. After this the client bundle
and the EntityManagerFactory saw different classes for the Entities.
I do not think this is Quiesce related though. If you have any ideas
about that would be great as I do not seem to find any solution.
>> The doc mentions "threads started by this bundle".
>>
>> If a bundle B calls a service from bundle A then the thread is not started
>> by bundle A. So it should be fine that this thread keeps running
>> even when the Activator.stop was called.
>>
> Here's what the spec says for services (section 5.7, osgi core 6): the
> behaviour of a a service object that becomes unregistered is undefined.
> Such service objects may continue to work properly or throw an exception at
> their discretion. Imho, in a production environment, you don't really want
> to rely on an undefined behavior. And if the service throws an exception,
> it's not a clean shutdown anymore.
I would interpret this part of the spec ("at their discretion") that the
service implementor is free to either continue working or throw an
exception.
So if he does not do anything special it should be fine.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-13 11:17 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
> On 13.02.2015 10:43, Guillaume Nodet wrote:
>
>>
>>
>> I think you did not understand the concept of *clean* shutdown which I'm
>> trying to explain.
>> I think there's a big difference between reacting to the fact that the
>> database connection has been lost, which is out of your control and the
>> only thing you can do is stop your own services asap, and cleanly shut
>> down
>> your application.
>>
>> Let's tale again my example where we have bundle B which is a service
>> exposed to the outside world and bundle A, a JPA layer. If I want to
>> update bundle A and I simply call bundleA.update(), this will cause users
>> of bundleB to have their calls fail. What we really need is the
>> following:
>> * stop any new users to use bundle B
>> * wait until all calls from bundle B to finish
>> * stop bundles A and B (which are not actively used anymore)
>> * update bundle A
>>
>> I think an update of bundleA should simply work without issues.
>
> From what I understood the following would happen.
> - bundleA stops which should not affect inflight processing inside bundle A
> - If bundle B uses blueprint then new calls to bundle A will block
> - bundleA is replaced and started
> - Calls from bundle B continue again
>
> So I think we do not have a problem in this case.
Your example expects a simple restart of the service, I'm talking about an
update/refresh of the bundle.
But even restarting the service, if the service is an EntityManager, that
does not work afaik : you can't just switch it in the middle of your
processing (what you've already done with the JDBC connection in the
context of your transaction can't be propagated to a new connection).
>
>
>
>
>>> This means that in order to stop A, I think you want to quiesce A and
>>> B.
>>>
>>>> One problem with blueprint, and it may be an implementation problem, is
>>>> the
>>>> following. I just made a test with the blueprint-testquiescebundle which
>>>> exposes a simple bean with a sleep method through blueprint. If I call
>>>> it
>>>> with a long sleep, and I concurrently stop the bundle, the bean will
>>>> continue executing while the bundle is already stopped. That's quite bad
>>>> actually.
>>>>
>>>> I think it is absolutely fine that threads of a bundle that stops or is
>>> even uninstalled continue to run. We might want to interrupt them so a
>>> sleep returns earlier but even if the sleep continues I do not see an
>>> immediate problem. It would just prevent the classloader from cleaning up
>>> the bundle classses for some time. Btw. I think having such a long sleep
>>> is
>>> an implementation problem that OSGi does not need to fix.
>>>
>>
>> Uh, no that's clearly not fine. BundleActivator#stop() says the following:
>>
>> /**
>> * Called when this bundle is stopped so the Framework can perform the
>> * bundle-specific activities necessary to stop the bundle. In general,
>> this
>> * method should undo the work that the {@code BundleActivator.start}
>> * method started. There should be no active threads that were started by
>> * this bundle when this bundle returns. A stopped bundle must not call
>> any
>> * Framework objects.
>> *
>> * <p>
>> * This method must complete and return to its caller in a timely manner.
>> */
>>
>> So, we can infer the following things:
>> * you're not allowed to wait for quite a long time to cleanly wait for
>> things to be ready to be stopped
>> * you're not allowed to leave threads running
>>
>> This leads to the conclusion that the OSGi api is not sufficient to cover
>> clean shutdown of bundles. I think the quiesce api aims to solve this
>> problem. I'm not saying it can't be improved or implemented differently,
>> I'm just trying to expose what I think is the purpose of this api.
>>
> The doc mentions "threads started by this bundle".
>
> If a bundle B calls a service from bundle A then the thread is not started
> by bundle A. So it should be fine that this thread keeps running
> even when the Activator.stop was called.
>
Here's what the spec says for services (section 5.7, osgi core 6): the
behaviour of a a service object that becomes unregistered is undefined.
Such service objects may continue to work properly or throw an exception at
their discretion. Imho, in a production environment, you don't really want
to rely on an undefined behavior. And if the service throws an exception,
it's not a clean shutdown anymore.
>
> So I think the OSGi API is fine and I still see no need for Quiesce.
> ... and honestly if the OSGi was not able to cover this quite typical case
> then it would be really bad and we should not use OSGi.
> If we are not sure then I propose we forward this question to the OSGi dev
> list and ask for some advice there.
Before doing that, we need to understand each other, and that's not the
case yet.
>
>
>
>>
>>
>> I agree, I'm not sure to fully understand it. That does not mean we
>> should
>> get rid of it just because we don't understand it ;-)
>>
> I agree. The problem is though if we have no one who understands it then
> we have a big problem when moving on.
>
> So my point of view is that we either have someone who understands it or
> at least have the use cases documented. If both is not the case then
> I indeed would rather like to get rid of Quiesce. Of course not in a bug
> fix or minor version. It should be ok for a major version though.
>
>
> Christian
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
On 13.02.2015 10:43, Guillaume Nodet wrote:
>
>
> I think you did not understand the concept of *clean* shutdown which I'm
> trying to explain.
> I think there's a big difference between reacting to the fact that the
> database connection has been lost, which is out of your control and the
> only thing you can do is stop your own services asap, and cleanly shut down
> your application.
>
> Let's tale again my example where we have bundle B which is a service
> exposed to the outside world and bundle A, a JPA layer. If I want to
> update bundle A and I simply call bundleA.update(), this will cause users
> of bundleB to have their calls fail. What we really need is the following:
> * stop any new users to use bundle B
> * wait until all calls from bundle B to finish
> * stop bundles A and B (which are not actively used anymore)
> * update bundle A
>
I think an update of bundleA should simply work without issues.
From what I understood the following would happen.
- bundleA stops which should not affect inflight processing inside bundle A
- If bundle B uses blueprint then new calls to bundle A will block
- bundleA is replaced and started
- Calls from bundle B continue again
So I think we do not have a problem in this case.
>>
>> This means that in order to stop A, I think you want to quiesce A and B.
>>> One problem with blueprint, and it may be an implementation problem, is the
>>> following. I just made a test with the blueprint-testquiescebundle which
>>> exposes a simple bean with a sleep method through blueprint. If I call it
>>> with a long sleep, and I concurrently stop the bundle, the bean will
>>> continue executing while the bundle is already stopped. That's quite bad
>>> actually.
>>>
>> I think it is absolutely fine that threads of a bundle that stops or is
>> even uninstalled continue to run. We might want to interrupt them so a
>> sleep returns earlier but even if the sleep continues I do not see an
>> immediate problem. It would just prevent the classloader from cleaning up
>> the bundle classses for some time. Btw. I think having such a long sleep is
>> an implementation problem that OSGi does not need to fix.
>
> Uh, no that's clearly not fine. BundleActivator#stop() says the following:
>
> /**
> * Called when this bundle is stopped so the Framework can perform the
> * bundle-specific activities necessary to stop the bundle. In general, this
> * method should undo the work that the {@code BundleActivator.start}
> * method started. There should be no active threads that were started by
> * this bundle when this bundle returns. A stopped bundle must not call any
> * Framework objects.
> *
> * <p>
> * This method must complete and return to its caller in a timely manner.
> */
>
> So, we can infer the following things:
> * you're not allowed to wait for quite a long time to cleanly wait for
> things to be ready to be stopped
> * you're not allowed to leave threads running
>
> This leads to the conclusion that the OSGi api is not sufficient to cover
> clean shutdown of bundles. I think the quiesce api aims to solve this
> problem. I'm not saying it can't be improved or implemented differently,
> I'm just trying to expose what I think is the purpose of this api.
The doc mentions "threads started by this bundle".
If a bundle B calls a service from bundle A then the thread is not
started by bundle A. So it should be fine that this thread keeps running
even when the Activator.stop was called.
So I think the OSGi API is fine and I still see no need for Quiesce.
... and honestly if the OSGi was not able to cover this quite typical
case then it would be really bad and we should not use OSGi.
If we are not sure then I propose we forward this question to the OSGi
dev list and ask for some advice there.
>
>
>
> I agree, I'm not sure to fully understand it. That does not mean we should
> get rid of it just because we don't understand it ;-)
I agree. The problem is though if we have no one who understands it then
we have a big problem when moving on.
So my point of view is that we either have someone who understands it or
at least have the use cases documented. If both is not the case then
I indeed would rather like to get rid of Quiesce. Of course not in a bug
fix or minor version. It should be ok for a major version though.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-13 9:04 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
> On 12.02.2015 21:41, Guillaume Nodet wrote:
>
>> Quiescing a single bundle does not make sense I think. Quiescing
>> obviously means you want to stop a bunch of bundles. In this case, if you
>> want to actually stop bundle A, you also need to stop all bundles using
>> services from A, so that would include B.
>>
> Why would I want to stop the other bundles? The situation that a service
> is going away is a normal thing in OSGi and the frameworks cover this
> situation.
>
> There are two cases:
> 1. Dynamic systems like DS
> All components using Services from A in a mandatory fashion will stop
> automatically. So no problem here.
>
> 2. Systems using damping like blueprint
> Blueprint will block calls to Services offered by A if there is no
> alternative service. This also seems to be fine for me. The calls will time
> out after some time and the callers will react according to their error
> handling functionality.
>
> So I do not see the need for anything better than what OSGi already offers
> for this use case.
I think you did not understand the concept of *clean* shutdown which I'm
trying to explain.
I think there's a big difference between reacting to the fact that the
database connection has been lost, which is out of your control and the
only thing you can do is stop your own services asap, and cleanly shut down
your application.
Let's tale again my example where we have bundle B which is a service
exposed to the outside world and bundle A, a JPA layer. If I want to
update bundle A and I simply call bundleA.update(), this will cause users
of bundleB to have their calls fail. What we really need is the following:
* stop any new users to use bundle B
* wait until all calls from bundle B to finish
* stop bundles A and B (which are not actively used anymore)
* update bundle A
>
>
> This means that in order to stop A, I think you want to quiesce A and B.
>> One problem with blueprint, and it may be an implementation problem, is the
>> following. I just made a test with the blueprint-testquiescebundle which
>> exposes a simple bean with a sleep method through blueprint. If I call it
>> with a long sleep, and I concurrently stop the bundle, the bean will
>> continue executing while the bundle is already stopped. That's quite bad
>> actually.
>>
> I think it is absolutely fine that threads of a bundle that stops or is
> even uninstalled continue to run. We might want to interrupt them so a
> sleep returns earlier but even if the sleep continues I do not see an
> immediate problem. It would just prevent the classloader from cleaning up
> the bundle classses for some time. Btw. I think having such a long sleep is
> an implementation problem that OSGi does not need to fix.
Uh, no that's clearly not fine. BundleActivator#stop() says the following:
/**
* Called when this bundle is stopped so the Framework can perform the
* bundle-specific activities necessary to stop the bundle. In general, this
* method should undo the work that the {@code BundleActivator.start}
* method started. There should be no active threads that were started by
* this bundle when this bundle returns. A stopped bundle must not call any
* Framework objects.
*
* <p>
* This method must complete and return to its caller in a timely manner.
*/
So, we can infer the following things:
* you're not allowed to wait for quite a long time to cleanly wait for
things to be ready to be stopped
* you're not allowed to leave threads running
This leads to the conclusion that the OSGi api is not sufficient to cover
clean shutdown of bundles. I think the quiesce api aims to solve this
problem. I'm not saying it can't be improved or implemented differently,
I'm just trying to expose what I think is the purpose of this api.
>
>
> I think the quiesce participant for blueprint aims to work around this
>> problem by making sure all calls to a service exposed using blueprint will
>> cause a delay in stopping the bundle. This can be worked around by adding a
>> destroy method to the bundle and synchronise all public methods. It could
>> also be done in a smarter way by adding a read/write lock instead. Anyway,
>> I'm not sure if that's the real problem, but the blueprint quiesce
>> participant actually does that.
>>
>
> Thanks for describing this. I think the main problem with Quiesce is that
> almost no one understands what problems it solves and if the current
> solution actually solves these problems.
>
I agree, I'm not sure to fully understand it. That does not mean we should
get rid of it just because we don't understand it ;-)
>
> So I think what we need is to take step back and describe what actual
> problems would occur without Quiesce and what Quiesce does to solve them.
> Only then we will be able to maintain the code and to improve it.
> The current status is that we have a proprietary API with no documentation
> anywhere and very little understanding by at least the active devs. So I
> hope some people from IBM can step up and help us to better understand this.
>
> I am pretty sure the current situation will lead to very bad quality of
> the code around Quiesce in mid term as people will either avoid to modify
> this code at all or the modifications will be wrong because of bad
> understanding of the design and the issues to solve.
Well, the first thing to do is to try understanding it, that's what I'm
trying to actually do.
>
>
>
> Christian
>
>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
On 12.02.2015 21:41, Guillaume Nodet wrote:
> Quiescing a single bundle does not make sense I think. Quiescing
> obviously means you want to stop a bunch of bundles. In this case, if
> you want to actually stop bundle A, you also need to stop all bundles
> using services from A, so that would include B.
Why would I want to stop the other bundles? The situation that a service
is going away is a normal thing in OSGi and the frameworks cover this
situation.
There are two cases:
1. Dynamic systems like DS
All components using Services from A in a mandatory fashion will stop
automatically. So no problem here.
2. Systems using damping like blueprint
Blueprint will block calls to Services offered by A if there is no
alternative service. This also seems to be fine for me. The calls will
time out after some time and the callers will react according to their
error handling functionality.
So I do not see the need for anything better than what OSGi already
offers for this use case.
> This means that in order to stop A, I think you want to quiesce A and
> B. One problem with blueprint, and it may be an implementation
> problem, is the following. I just made a test with the
> blueprint-testquiescebundle which exposes a simple bean with a sleep
> method through blueprint. If I call it with a long sleep, and I
> concurrently stop the bundle, the bean will continue executing while
> the bundle is already stopped. That's quite bad actually.
I think it is absolutely fine that threads of a bundle that stops or is
even uninstalled continue to run. We might want to interrupt them so a
sleep returns earlier but even if the sleep continues I do not see an
immediate problem. It would just prevent the classloader from cleaning
up the bundle classses for some time. Btw. I think having such a long
sleep is an implementation problem that OSGi does not need to fix.
> I think the quiesce participant for blueprint aims to work around this
> problem by making sure all calls to a service exposed using blueprint
> will cause a delay in stopping the bundle. This can be worked around
> by adding a destroy method to the bundle and synchronise all public
> methods. It could also be done in a smarter way by adding a read/write
> lock instead. Anyway, I'm not sure if that's the real problem, but the
> blueprint quiesce participant actually does that.
Thanks for describing this. I think the main problem with Quiesce is
that almost no one understands what problems it solves and if the
current solution actually solves these problems.
So I think what we need is to take step back and describe what actual
problems would occur without Quiesce and what Quiesce does to solve
them. Only then we will be able to maintain the code and to improve it.
The current status is that we have a proprietary API with no
documentation anywhere and very little understanding by at least the
active devs. So I hope some people from IBM can step up and help us to
better understand this.
I am pretty sure the current situation will lead to very bad quality of
the code around Quiesce in mid term as people will either avoid to
modify this code at all or the modifications will be wrong because of
bad understanding of the design and the issues to solve.
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-12 12:21 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
> I think this is in part because of the service damping in blueprint.
> Blueprint will not shut down a service if a service it depends on goes away.
> If you have a more reactive framework like DS it would not happen. If you
> remove the DataSource then any DS component using it will also go down.
>
> Besides I am not sure how quiesce can solve that? You still have to
> quiesce the correct bundle.
> Imagine bundle A contains the bean with JPA access and bundle B contains
> the CXF service which uses the service from B.
> If you quiesce bundle A then bundle B will still provide the CXF service
> but does not work anymore (when using blueprint).
>
> It would have the same effect to simple stop bundle A. Maybe I am missing
> something but how does the Quiesce API support this better?
Quiescing a single bundle does not make sense I think. Quiescing obviously
means you want to stop a bunch of bundles.
In this case, if you want to actually stop bundle A, you also need to stop
all bundles using services from A, so that would include B.
This means that in order to stop A, I think you want to quiesce A and B.
One problem with blueprint, and it may be an implementation problem, is the
following.
I just made a test with the blueprint-testquiescebundle which exposes a
simple bean with a sleep method through blueprint. If I call it with a
long sleep, and I concurrently stop the bundle, the bean will continue
executing while the bundle is already stopped. That's quite bad actually.
I think the quiesce participant for blueprint aims to work around this
problem by making sure all calls to a service exposed using blueprint will
cause a delay in stopping the bundle.
This can be worked around by adding a destroy method to the bundle and
synchronise all public methods. It could also be done in a smarter way by
adding a read/write lock instead.
Anyway, I'm not sure if that's the real problem, but the blueprint quiesce
participant actually does that.
>
>
> Christian
>
> On 12.02.2015 12:06, Guillaume Nodet wrote:
>
>> I think I actually do understand the need.
>> When you want to stop a bunch of bundles, you can choose to stop them in a
>> random order, or in a smart way.
>> At the end the result may not be the same.
>>
>> Let me take an example: a cxf service which accesses a database.
>> If you choose a random order, you may shut down the database access
>> (DataSource in osgi for example), before shutting down the cxf service.
>> The effect will be that you will make inflight requests to fail.
>> The correct way would be to shut down the cxf service in a clean way, i.e.
>> do not accept any more incoming requests, and wait for inflight requests
>> to
>> be finished, then shut down the cxf service, then shutdown the jdbc
>> factory.
>>
>> This is somewhat what the bluprint extender try to do when you shutdown
>> the
>> extender itself, but if you bundles encompass multiple different
>> extenders,
>> they need to cooperate in a smarter way. This could also apply to CXF
>> services, camel routes, etc...
>>
>>
>>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
I think this is in part because of the service damping in blueprint.
Blueprint will not shut down a service if a service it depends on goes away.
If you have a more reactive framework like DS it would not happen. If
you remove the DataSource then any DS component using it will also go down.
Besides I am not sure how quiesce can solve that? You still have to
quiesce the correct bundle.
Imagine bundle A contains the bean with JPA access and bundle B contains
the CXF service which uses the service from B.
If you quiesce bundle A then bundle B will still provide the CXF service
but does not work anymore (when using blueprint).
It would have the same effect to simple stop bundle A. Maybe I am
missing something but how does the Quiesce API support this better?
Christian
On 12.02.2015 12:06, Guillaume Nodet wrote:
> I think I actually do understand the need.
> When you want to stop a bunch of bundles, you can choose to stop them in a
> random order, or in a smart way.
> At the end the result may not be the same.
>
> Let me take an example: a cxf service which accesses a database.
> If you choose a random order, you may shut down the database access
> (DataSource in osgi for example), before shutting down the cxf service.
> The effect will be that you will make inflight requests to fail.
> The correct way would be to shut down the cxf service in a clean way, i.e.
> do not accept any more incoming requests, and wait for inflight requests to
> be finished, then shut down the cxf service, then shutdown the jdbc factory.
>
> This is somewhat what the bluprint extender try to do when you shutdown the
> extender itself, but if you bundles encompass multiple different extenders,
> they need to cooperate in a smarter way. This could also apply to CXF
> services, camel routes, etc...
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
I think I actually do understand the need.
When you want to stop a bunch of bundles, you can choose to stop them in a
random order, or in a smart way.
At the end the result may not be the same.
Let me take an example: a cxf service which accesses a database.
If you choose a random order, you may shut down the database access
(DataSource in osgi for example), before shutting down the cxf service.
The effect will be that you will make inflight requests to fail.
The correct way would be to shut down the cxf service in a clean way, i.e.
do not accept any more incoming requests, and wait for inflight requests to
be finished, then shut down the cxf service, then shutdown the jdbc factory.
This is somewhat what the bluprint extender try to do when you shutdown the
extender itself, but if you bundles encompass multiple different extenders,
they need to cooperate in a smarter way. This could also apply to CXF
services, camel routes, etc...
2015-02-12 11:55 GMT+01:00 Christian Schneider <ch...@die-schneider.net>:
> I understand the need to clean up but the OSGi BundleTracker should
> already provide all that.
> We even already track these events in aries jpa.
> See http://www.osgi.org/javadoc/r4v42/org/osgi/util/tracker/
> BundleTracker.html
>
> So when a bundle is stopped or uninstalled then aries jpa can already
> clean up without the Quiesce API.
> Interestingly in apache karaf we do not use the Quiesce API.
>
> Can you describe what happens when we just uninstall a persisence unit
> bundle? Will the resources then not be cleaned up correctly?
>
> Btw. One of our customers is facing this issue https://issues.apache.org/
> jira/browse/ARIES-1270
> Is this one of these problematic cases or is it just a bug that we can fix
> independently of the Quiesce API? The issue seems to be related to the
> jpa.container module as
> it is independent of jpa.container.context. Unfortunately I was not able
> to find the root cause of it.
>
> Christian
>
> On 12.02.2015 11:25, Graham Charters wrote:
>
>> Hi David/Christian,
>>
>> Here is what Mark wrote, that didn't make it:
>>
>> "Quiesce is about stopping a bundle gracefully prior to its
>> uninstallation,
>> which is why we've had no need to implement a 'restart' capability. We
>> currently use Apache Aries quiesce within IBM WebSphere Application
>> Server.
>> It's used in a scenario where an administrator is replacing some of the
>> bundles within a running application. The code has been in use in that
>> product for some years now so yes, I'm sorry but we do have a definite
>> need
>> for the facility."
>>
>> Essentially, we use it to notify containers/extenders of bundles that are
>> about to be uninstalled (prior to that actually being done) so they can
>> perform house-keeping ahead of uninstallation being kicked off.
>>
>> Regards, Graham.
>>
>>
>>
>>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
I understand the need to clean up but the OSGi BundleTracker should
already provide all that.
We even already track these events in aries jpa.
See
http://www.osgi.org/javadoc/r4v42/org/osgi/util/tracker/BundleTracker.html
So when a bundle is stopped or uninstalled then aries jpa can already
clean up without the Quiesce API.
Interestingly in apache karaf we do not use the Quiesce API.
Can you describe what happens when we just uninstall a persisence unit
bundle? Will the resources then not be cleaned up correctly?
Btw. One of our customers is facing this issue
https://issues.apache.org/jira/browse/ARIES-1270
Is this one of these problematic cases or is it just a bug that we can
fix independently of the Quiesce API? The issue seems to be related to
the jpa.container module as
it is independent of jpa.container.context. Unfortunately I was not able
to find the root cause of it.
Christian
On 12.02.2015 11:25, Graham Charters wrote:
> Hi David/Christian,
>
> Here is what Mark wrote, that didn't make it:
>
> "Quiesce is about stopping a bundle gracefully prior to its uninstallation,
> which is why we've had no need to implement a 'restart' capability. We
> currently use Apache Aries quiesce within IBM WebSphere Application Server.
> It's used in a scenario where an administrator is replacing some of the
> bundles within a running application. The code has been in use in that
> product for some years now so yes, I'm sorry but we do have a definite need
> for the facility."
>
> Essentially, we use it to notify containers/extenders of bundles that are
> about to be uninstalled (prior to that actually being done) so they can
> perform house-keeping ahead of uninstallation being kicked off.
>
> Regards, Graham.
>
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Guillaume Nodet <gn...@apache.org>.
2015-02-12 17:33 GMT+01:00 David Bosschaert <da...@gmail.com>:
> I would definitely be interested in seeing alternatives. I still don't
> fully understand the use-case to be honest. Looking at the database
> example from Guillaume above, I don't see why the bundles need to be
> taken down in a controlled manner. Let's say the database bundle is
> stopped before the CXF bundle, then surely the CXF bundle can react to
> that in the appropriate manner. In OSGi services are dynamic, they can
> come and go at any point in time. If that CXF bundle has a dependency
> on the database service it should handle the case when that database
> service goes away, either by using DS, Blueprint or something
> lower-level like a ServiceTracker or ServiceListener. The service
> might re-appear at a later point in time and again the using bundle
> should be able to deal with that.
>
I disagree. What is needed is a clean shutdown of bundles.
The only way to achieve that without taking control of the order in which
bundles are stopped would be by intercepting the ServiceEvent.UNREGISTERING
event and *blocking* the call. Though the problem is that you still can't
make the difference between a clean shutdown and an "immediate" shutdown.
Saying that the extender should take care of the case where the database
goes away is correct, but not sufficient. Obviously, the system needs to
take into account things like network connections going down. But throwing
an exception to clients and rolling back all inflight transactions is just
not something you want when updating a bundle on a production system.
> I think the code cleanup estimate from Christian is impressive. If we
> have an alternative to compare the Quiesce functionality with I think
> this would be interesting. I don't mean to say remove the Quiesce
> support, but I think it would be healthy to have the discussion about
> alternatives and compare options.
>
I agree. But if you think the use case I outlined are invalid, we're
starting on the wrong foot ;-)
>
> Best regards,
>
> David
>
> On 12 February 2015 at 16:20, Christian Schneider
> <ch...@die-schneider.net> wrote:
> > I just checked the diff of my changes when removing Quiesce. We would be
> > able to remove ~20 classes and 1500 lines of code.
> > So I think it would be worth the time to check if we can achieve the same
> > with an alternative solution.
> >
> > Apart from complexity and code size my big problem with the way Quiesce
> > works in aries jpa is that it creates a proxy for the EMF and the EM with
> > artificial methods to shut it down.
> > I think that is not a good idea at all. With the proxied EMF we also shut
> > down EMs that other bundles created. This can lead to problems in these
> > bundles as they are probably not aware of this.
> >
> > I think we should rather track the EMs we create ourself (mainly in the
> jpa
> > blueprint code) and close these when the quiesce happens.
> >
> > Christian
> >
> > On 12.02.2015 11:25, Graham Charters wrote:
> >>
> >> Hi David/Christian,
> >>
> >> Here is what Mark wrote, that didn't make it:
> >>
> >> "Quiesce is about stopping a bundle gracefully prior to its
> >> uninstallation,
> >> which is why we've had no need to implement a 'restart' capability. We
> >> currently use Apache Aries quiesce within IBM WebSphere Application
> >> Server.
> >> It's used in a scenario where an administrator is replacing some of the
> >> bundles within a running application. The code has been in use in that
> >> product for some years now so yes, I'm sorry but we do have a definite
> >> need
> >> for the facility."
> >>
> >> Essentially, we use it to notify containers/extenders of bundles that
> are
> >> about to be uninstalled (prior to that actually being done) so they can
> >> perform house-keeping ahead of uninstallation being kicked off.
> >>
> >> Regards, Graham.
> >>
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
>
Re: Do we really need Quiesce support?
Posted by David Bosschaert <da...@gmail.com>.
I would definitely be interested in seeing alternatives. I still don't
fully understand the use-case to be honest. Looking at the database
example from Guillaume above, I don't see why the bundles need to be
taken down in a controlled manner. Let's say the database bundle is
stopped before the CXF bundle, then surely the CXF bundle can react to
that in the appropriate manner. In OSGi services are dynamic, they can
come and go at any point in time. If that CXF bundle has a dependency
on the database service it should handle the case when that database
service goes away, either by using DS, Blueprint or something
lower-level like a ServiceTracker or ServiceListener. The service
might re-appear at a later point in time and again the using bundle
should be able to deal with that.
I think the code cleanup estimate from Christian is impressive. If we
have an alternative to compare the Quiesce functionality with I think
this would be interesting. I don't mean to say remove the Quiesce
support, but I think it would be healthy to have the discussion about
alternatives and compare options.
Best regards,
David
On 12 February 2015 at 16:20, Christian Schneider
<ch...@die-schneider.net> wrote:
> I just checked the diff of my changes when removing Quiesce. We would be
> able to remove ~20 classes and 1500 lines of code.
> So I think it would be worth the time to check if we can achieve the same
> with an alternative solution.
>
> Apart from complexity and code size my big problem with the way Quiesce
> works in aries jpa is that it creates a proxy for the EMF and the EM with
> artificial methods to shut it down.
> I think that is not a good idea at all. With the proxied EMF we also shut
> down EMs that other bundles created. This can lead to problems in these
> bundles as they are probably not aware of this.
>
> I think we should rather track the EMs we create ourself (mainly in the jpa
> blueprint code) and close these when the quiesce happens.
>
> Christian
>
> On 12.02.2015 11:25, Graham Charters wrote:
>>
>> Hi David/Christian,
>>
>> Here is what Mark wrote, that didn't make it:
>>
>> "Quiesce is about stopping a bundle gracefully prior to its
>> uninstallation,
>> which is why we've had no need to implement a 'restart' capability. We
>> currently use Apache Aries quiesce within IBM WebSphere Application
>> Server.
>> It's used in a scenario where an administrator is replacing some of the
>> bundles within a running application. The code has been in use in that
>> product for some years now so yes, I'm sorry but we do have a definite
>> need
>> for the facility."
>>
>> Essentially, we use it to notify containers/extenders of bundles that are
>> about to be uninstalled (prior to that actually being done) so they can
>> perform house-keeping ahead of uninstallation being kicked off.
>>
>> Regards, Graham.
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
I just checked the diff of my changes when removing Quiesce. We would be
able to remove ~20 classes and 1500 lines of code.
So I think it would be worth the time to check if we can achieve the
same with an alternative solution.
Apart from complexity and code size my big problem with the way Quiesce
works in aries jpa is that it creates a proxy for the EMF and the EM
with artificial methods to shut it down.
I think that is not a good idea at all. With the proxied EMF we also
shut down EMs that other bundles created. This can lead to problems in
these bundles as they are probably not aware of this.
I think we should rather track the EMs we create ourself (mainly in the
jpa blueprint code) and close these when the quiesce happens.
Christian
On 12.02.2015 11:25, Graham Charters wrote:
> Hi David/Christian,
>
> Here is what Mark wrote, that didn't make it:
>
> "Quiesce is about stopping a bundle gracefully prior to its uninstallation,
> which is why we've had no need to implement a 'restart' capability. We
> currently use Apache Aries quiesce within IBM WebSphere Application Server.
> It's used in a scenario where an administrator is replacing some of the
> bundles within a running application. The code has been in use in that
> product for some years now so yes, I'm sorry but we do have a definite need
> for the facility."
>
> Essentially, we use it to notify containers/extenders of bundles that are
> about to be uninstalled (prior to that actually being done) so they can
> perform house-keeping ahead of uninstallation being kicked off.
>
> Regards, Graham.
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com
Re: Do we really need Quiesce support?
Posted by Graham Charters <gc...@gmail.com>.
Hi David/Christian,
Here is what Mark wrote, that didn't make it:
"Quiesce is about stopping a bundle gracefully prior to its uninstallation,
which is why we've had no need to implement a 'restart' capability. We
currently use Apache Aries quiesce within IBM WebSphere Application Server.
It's used in a scenario where an administrator is replacing some of the
bundles within a running application. The code has been in use in that
product for some years now so yes, I'm sorry but we do have a definite need
for the facility."
Essentially, we use it to notify containers/extenders of bundles that are
about to be uninstalled (prior to that actually being done) so they can
perform house-keeping ahead of uninstallation being kicked off.
Regards, Graham.
On 12 February 2015 at 10:12, David Bosschaert <da...@gmail.com>
wrote:
> I'd like to understand what the use-case for the Quiesce functionality
> actually is. Graham, would you like to explain the actual benefit that
> it provides?
> Once we understand that better, maybe we can think of a structure that
> maintains Quiesce support with less complication in the code...
>
> Cheers,
>
> David
>
> On 12 February 2015 at 10:30, Graham Charters <gc...@gmail.com> wrote:
> > Hi Christian,
> >
> > Apologies for not responding sooner, Mark Nuttall tried to reply on this
> > thread, but his email did not make it. We use Quiesce support in
> WebSphere
> > so need this to be left in. Please do not remove it. Many thanks.
> >
> > Graham.
> >
> > On 12 February 2015 at 08:02, Christian Schneider <
> chris@die-schneider.net>
> > wrote:
> >
> >> I created an issue to track the removal of the quiesce support in aries
> >> jpa.
> >> https://issues.apache.org/jira/browse/ARIES-1297
> >>
> >> I have prepared a commit and plan to apply the removal on next monday if
> >> no one opposes.
> >>
> >> Christian
> >>
> >>
> >> On 10.02.2015 09:15, Christian Schneider wrote:
> >>
> >>> In several places in aries we support the Quiesce API. As far as I
> >>> understood its purpose is to shut down bundles without actually
> stopping
> >>> them. It does not seem to be any standard as the interfaces are in the
> >>> Aries namespace.
> >>>
> >>> The problem with it is that it complicates the code a lot. For example
> in
> >>> jpa container we create a proxy for each EntityManager that also
> supports
> >>> quiesce methods even though they are
> >>> not part of the EntityManager interface. The code in other modules is
> >>> similarly affected.
> >>>
> >>> Besides this I do not see anyone using the API. I am also not sure why
> we
> >>> need it at all as OSGi already has nice lifecycle methods to stop
> bundles
> >>> which should have a similar effect if implemented nicely. One other
> >>> disadvantge of Quiesce is that you can only shut down a bundle. You
> can not
> >>> restart it again.
> >>>
> >>> So the question is: Do we really need to support the Quiesce API? The
> >>> code would be a lot less complex and error prone without it.
> >>>
> >>> WDYT?
> >>>
> >>> Christian
> >>>
> >>>
> >>>
> >>
> >> --
> >> Christian Schneider
> >> http://www.liquid-reality.de
> >>
> >> Open Source Architect
> >> http://www.talend.com
> >>
> >>
>
Re: Do we really need Quiesce support?
Posted by David Bosschaert <da...@gmail.com>.
I'd like to understand what the use-case for the Quiesce functionality
actually is. Graham, would you like to explain the actual benefit that
it provides?
Once we understand that better, maybe we can think of a structure that
maintains Quiesce support with less complication in the code...
Cheers,
David
On 12 February 2015 at 10:30, Graham Charters <gc...@gmail.com> wrote:
> Hi Christian,
>
> Apologies for not responding sooner, Mark Nuttall tried to reply on this
> thread, but his email did not make it. We use Quiesce support in WebSphere
> so need this to be left in. Please do not remove it. Many thanks.
>
> Graham.
>
> On 12 February 2015 at 08:02, Christian Schneider <ch...@die-schneider.net>
> wrote:
>
>> I created an issue to track the removal of the quiesce support in aries
>> jpa.
>> https://issues.apache.org/jira/browse/ARIES-1297
>>
>> I have prepared a commit and plan to apply the removal on next monday if
>> no one opposes.
>>
>> Christian
>>
>>
>> On 10.02.2015 09:15, Christian Schneider wrote:
>>
>>> In several places in aries we support the Quiesce API. As far as I
>>> understood its purpose is to shut down bundles without actually stopping
>>> them. It does not seem to be any standard as the interfaces are in the
>>> Aries namespace.
>>>
>>> The problem with it is that it complicates the code a lot. For example in
>>> jpa container we create a proxy for each EntityManager that also supports
>>> quiesce methods even though they are
>>> not part of the EntityManager interface. The code in other modules is
>>> similarly affected.
>>>
>>> Besides this I do not see anyone using the API. I am also not sure why we
>>> need it at all as OSGi already has nice lifecycle methods to stop bundles
>>> which should have a similar effect if implemented nicely. One other
>>> disadvantge of Quiesce is that you can only shut down a bundle. You can not
>>> restart it again.
>>>
>>> So the question is: Do we really need to support the Quiesce API? The
>>> code would be a lot less complex and error prone without it.
>>>
>>> WDYT?
>>>
>>> Christian
>>>
>>>
>>>
>>
>> --
>> Christian Schneider
>> http://www.liquid-reality.de
>>
>> Open Source Architect
>> http://www.talend.com
>>
>>
Re: Do we really need Quiesce support?
Posted by Graham Charters <gc...@gmail.com>.
Hi Christian,
Apologies for not responding sooner, Mark Nuttall tried to reply on this
thread, but his email did not make it. We use Quiesce support in WebSphere
so need this to be left in. Please do not remove it. Many thanks.
Graham.
On 12 February 2015 at 08:02, Christian Schneider <ch...@die-schneider.net>
wrote:
> I created an issue to track the removal of the quiesce support in aries
> jpa.
> https://issues.apache.org/jira/browse/ARIES-1297
>
> I have prepared a commit and plan to apply the removal on next monday if
> no one opposes.
>
> Christian
>
>
> On 10.02.2015 09:15, Christian Schneider wrote:
>
>> In several places in aries we support the Quiesce API. As far as I
>> understood its purpose is to shut down bundles without actually stopping
>> them. It does not seem to be any standard as the interfaces are in the
>> Aries namespace.
>>
>> The problem with it is that it complicates the code a lot. For example in
>> jpa container we create a proxy for each EntityManager that also supports
>> quiesce methods even though they are
>> not part of the EntityManager interface. The code in other modules is
>> similarly affected.
>>
>> Besides this I do not see anyone using the API. I am also not sure why we
>> need it at all as OSGi already has nice lifecycle methods to stop bundles
>> which should have a similar effect if implemented nicely. One other
>> disadvantge of Quiesce is that you can only shut down a bundle. You can not
>> restart it again.
>>
>> So the question is: Do we really need to support the Quiesce API? The
>> code would be a lot less complex and error prone without it.
>>
>> WDYT?
>>
>> Christian
>>
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: Do we really need Quiesce support?
Posted by Christian Schneider <ch...@die-schneider.net>.
I created an issue to track the removal of the quiesce support in aries jpa.
https://issues.apache.org/jira/browse/ARIES-1297
I have prepared a commit and plan to apply the removal on next monday if
no one opposes.
Christian
On 10.02.2015 09:15, Christian Schneider wrote:
> In several places in aries we support the Quiesce API. As far as I
> understood its purpose is to shut down bundles without actually
> stopping them. It does not seem to be any standard as the interfaces
> are in the Aries namespace.
>
> The problem with it is that it complicates the code a lot. For example
> in jpa container we create a proxy for each EntityManager that also
> supports quiesce methods even though they are
> not part of the EntityManager interface. The code in other modules is
> similarly affected.
>
> Besides this I do not see anyone using the API. I am also not sure why
> we need it at all as OSGi already has nice lifecycle methods to stop
> bundles which should have a similar effect if implemented nicely. One
> other disadvantge of Quiesce is that you can only shut down a bundle.
> You can not restart it again.
>
> So the question is: Do we really need to support the Quiesce API? The
> code would be a lot less complex and error prone without it.
>
> WDYT?
>
> Christian
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com