You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@felix.apache.org by Guillaume Nodet <gn...@gmail.com> on 2012/05/24 12:09:19 UTC
Deadlock with ManagedServiceFactory
I have the following deadlock that sometimes happen:
Thread 1:
start the bundle
register a ManagedServiceFactory
Thread 2:
stop the bundle
grab the bundle lock
try to destroy the ManagedServiceFactory
deadlock on grabbing the ManagedServiceFactory lock
Thread 3:
in a different thread, the ConfigAdmin will call the
ManagedServiceFactory#update()
enter synchronized block in the ManagedServiceFactory
register a service
try to grab the bundle lock
I don't think the problem comes from my ManagedServiceFactory, as it
has to be synchronized in order for the destruction of the
ManagedServiceFactory to make sure we destroy all the previously
created services.
It seems to me that the problem comes from FELIX-3082 which allows the
registration of services while the bundle is stopping.
I think if we remove that bit, the third thread will reject the
service registration, exit the ManagedServiceFactory#update() and
release the ManagedServiceFactory lock.
I'll give it a try, but thoughts are welcomed.
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Guillaume Nodet <gn...@gmail.com>.
Well, lenient can be really bad when you use system resources such as
sockets, or file locks.
My problems happen when updating/refrehsing the bundle which holds
such resources, so I really need them to be freed when the activator
is stopped.
I'll have a closer look at what you suggests, thx.
On Thu, May 24, 2012 at 3:37 PM, Felix Meschberger <fm...@adobe.com> wrote:
> Hi,
>
> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>
>> The assumptions are right.
>>
>> Do you have any pointer to a well written ManagedServiceFactory that
>> can make sure calls to updates are finished before destroying the
>> ManagedServiceFactory without any synchronized blocks ?
>
> Not, off the top of my head.
>
> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>
> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>
> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>
> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>
> Regards
> Felix
>
>>
>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>> Hi,
>>>
>>>
>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>
>>>> I have the following deadlock that sometimes happen:
>>>>
>>>> Thread 1:
>>>
>>>> start the bundle
>>>> register a ManagedServiceFactory
>>>
>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>
>>>>
>>>> Thread 2:
>>>> stop the bundle
>>>
>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>
>>>> grab the bundle lock
>>>> try to destroy the ManagedServiceFactory
>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>
>>>> Thread 3:
>>>
>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>
>>>> in a different thread, the ConfigAdmin will call the
>>>> ManagedServiceFactory#update()
>>>> enter synchronized block in the ManagedServiceFactory
>>>> register a service
>>>> try to grab the bundle lock
>>>>
>>>>
>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>> has to be synchronized in order for the destruction of the
>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>> created services.
>>>
>>> I disagree.
>>>
>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>
>>> Regards
>>> Felix
>>>
>>>>
>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>> registration of services while the bundle is stopping.
>>>> I think if we remove that bit, the third thread will reject the
>>>> service registration, exit the ManagedServiceFactory#update() and
>>>> release the ManagedServiceFactory lock.
>>>>
>>>> I'll give it a try, but thoughts are welcomed.
>>>>
>>>> --
>>>> ------------------------
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> FuseSource, Integration everywhere
>>>> http://fusesource.com
>>>
>>
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com
>
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Guillaume Nodet <gn...@gmail.com>.
Right.
I suppose I can enhance my BaseManagedServiceFactory the following way
https://gist.github.com/2786874
It will wait for a certain amount of time before interrupting the
threads, which should get rid of the deadlock in case it happens.
I think I'll put that class in the utils project so that other people
can benefit from it.
On Fri, May 25, 2012 at 10:57 AM, Felix Meschberger <fm...@adobe.com> wrote:
> Hi,
>
> Am 24.05.2012 um 18:18 schrieb Guillaume Nodet:
>
>> ... which makes me thing that being able to register services (or at
>> least try) while the bundle is stopping is still wrong.
>> What's the use case for that ?
>
> I agree it is arguable, but the core spec states in Section 5.2.3, Registering Services :
>
>> The Framework permits bundles to register and unregister service objects dynamically. Therefore, a bundle is permitted to register service objects at any time during the STARTING, ACTIVE or STOPPING states.
>
> So, it must be possible.
>
> Regards
> Felix
>
>
>>
>> On Thu, May 24, 2012 at 6:05 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>>> That's actually a really good idea.
>>> For the record, I came up with the following class
>>> https://gist.github.com/2782423
>>> which can't prevent all deadlocks (I think there's still a small
>>> window of time where the service registration could be attempted while
>>> the bundle lock is grabbed by the framework when stopping the bundle,
>>> but before the factory destroy() method is called), but it's way
>>> better.
>>> Though, the mere fact I don't see a nice way to avoid the deadlock
>>> kinda bothers me.
>>>
>>> On Thu, May 24, 2012 at 3:58 PM, Marcel Offermans
>>> <ma...@luminis.nl> wrote:
>>>> Another pattern you could use here is to have the updated method only validate if the configuration it received is valid and not process it at all in the updated method but delegate that to a different method. That way, you can probably even prevent the creation of the objects you mention in your example altogether. Of course there then has to be "something else" to delegate to (like some other thread).
>>>>
>>>> Greetings, Marcel
>>>>
>>>>
>>>> On May 24, 2012, at 15:37 PM, Felix Meschberger wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>>>>>
>>>>>> The assumptions are right.
>>>>>>
>>>>>> Do you have any pointer to a well written ManagedServiceFactory that
>>>>>> can make sure calls to updates are finished before destroying the
>>>>>> ManagedServiceFactory without any synchronized blocks ?
>>>>>
>>>>> Not, off the top of my head.
>>>>>
>>>>> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>>>>>
>>>>> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>>>>>
>>>>> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>>>>>
>>>>> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>>>>>
>>>>> Regards
>>>>> Felix
>>>>>
>>>>>>
>>>>>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>>>>>
>>>>>>>> I have the following deadlock that sometimes happen:
>>>>>>>>
>>>>>>>> Thread 1:
>>>>>>>
>>>>>>>> start the bundle
>>>>>>>> register a ManagedServiceFactory
>>>>>>>
>>>>>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>>>>>
>>>>>>>>
>>>>>>>> Thread 2:
>>>>>>>> stop the bundle
>>>>>>>
>>>>>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>>>>>
>>>>>>>> grab the bundle lock
>>>>>>>> try to destroy the ManagedServiceFactory
>>>>>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>>>>>
>>>>>>>> Thread 3:
>>>>>>>
>>>>>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>>>>>
>>>>>>>> in a different thread, the ConfigAdmin will call the
>>>>>>>> ManagedServiceFactory#update()
>>>>>>>> enter synchronized block in the ManagedServiceFactory
>>>>>>>> register a service
>>>>>>>> try to grab the bundle lock
>>>>>>>>
>>>>>>>>
>>>>>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>>>>>> has to be synchronized in order for the destruction of the
>>>>>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>>>>>> created services.
>>>>>>>
>>>>>>> I disagree.
>>>>>>>
>>>>>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>>>>>
>>>>>>> Regards
>>>>>>> Felix
>>>>>>>
>>>>>>>>
>>>>>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>>>>>> registration of services while the bundle is stopping.
>>>>>>>> I think if we remove that bit, the third thread will reject the
>>>>>>>> service registration, exit the ManagedServiceFactory#update() and
>>>>>>>> release the ManagedServiceFactory lock.
>>>>>>>>
>>>>>>>> I'll give it a try, but thoughts are welcomed.
>>>>>>>>
>>>>>>>> --
>>>>>>>> ------------------------
>>>>>>>> Guillaume Nodet
>>>>>>>> ------------------------
>>>>>>>> Blog: http://gnodet.blogspot.com/
>>>>>>>> ------------------------
>>>>>>>> FuseSource, Integration everywhere
>>>>>>>> http://fusesource.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ------------------------
>>>>>> Guillaume Nodet
>>>>>> ------------------------
>>>>>> Blog: http://gnodet.blogspot.com/
>>>>>> ------------------------
>>>>>> FuseSource, Integration everywhere
>>>>>> http://fusesource.com
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> ------------------------
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>> ------------------------
>>> FuseSource, Integration everywhere
>>> http://fusesource.com
>>
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com
>
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Felix Meschberger <fm...@adobe.com>.
Hi,
Am 24.05.2012 um 18:18 schrieb Guillaume Nodet:
> ... which makes me thing that being able to register services (or at
> least try) while the bundle is stopping is still wrong.
> What's the use case for that ?
I agree it is arguable, but the core spec states in Section 5.2.3, Registering Services :
> The Framework permits bundles to register and unregister service objects dynamically. Therefore, a bundle is permitted to register service objects at any time during the STARTING, ACTIVE or STOPPING states.
So, it must be possible.
Regards
Felix
>
> On Thu, May 24, 2012 at 6:05 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>> That's actually a really good idea.
>> For the record, I came up with the following class
>> https://gist.github.com/2782423
>> which can't prevent all deadlocks (I think there's still a small
>> window of time where the service registration could be attempted while
>> the bundle lock is grabbed by the framework when stopping the bundle,
>> but before the factory destroy() method is called), but it's way
>> better.
>> Though, the mere fact I don't see a nice way to avoid the deadlock
>> kinda bothers me.
>>
>> On Thu, May 24, 2012 at 3:58 PM, Marcel Offermans
>> <ma...@luminis.nl> wrote:
>>> Another pattern you could use here is to have the updated method only validate if the configuration it received is valid and not process it at all in the updated method but delegate that to a different method. That way, you can probably even prevent the creation of the objects you mention in your example altogether. Of course there then has to be "something else" to delegate to (like some other thread).
>>>
>>> Greetings, Marcel
>>>
>>>
>>> On May 24, 2012, at 15:37 PM, Felix Meschberger wrote:
>>>
>>>> Hi,
>>>>
>>>> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>>>>
>>>>> The assumptions are right.
>>>>>
>>>>> Do you have any pointer to a well written ManagedServiceFactory that
>>>>> can make sure calls to updates are finished before destroying the
>>>>> ManagedServiceFactory without any synchronized blocks ?
>>>>
>>>> Not, off the top of my head.
>>>>
>>>> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>>>>
>>>> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>>>>
>>>> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>>>>
>>>> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>>>>
>>>> Regards
>>>> Felix
>>>>
>>>>>
>>>>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>>>>
>>>>>>> I have the following deadlock that sometimes happen:
>>>>>>>
>>>>>>> Thread 1:
>>>>>>
>>>>>>> start the bundle
>>>>>>> register a ManagedServiceFactory
>>>>>>
>>>>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>>>>
>>>>>>>
>>>>>>> Thread 2:
>>>>>>> stop the bundle
>>>>>>
>>>>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>>>>
>>>>>>> grab the bundle lock
>>>>>>> try to destroy the ManagedServiceFactory
>>>>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>>>>
>>>>>>> Thread 3:
>>>>>>
>>>>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>>>>
>>>>>>> in a different thread, the ConfigAdmin will call the
>>>>>>> ManagedServiceFactory#update()
>>>>>>> enter synchronized block in the ManagedServiceFactory
>>>>>>> register a service
>>>>>>> try to grab the bundle lock
>>>>>>>
>>>>>>>
>>>>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>>>>> has to be synchronized in order for the destruction of the
>>>>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>>>>> created services.
>>>>>>
>>>>>> I disagree.
>>>>>>
>>>>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>>>>
>>>>>> Regards
>>>>>> Felix
>>>>>>
>>>>>>>
>>>>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>>>>> registration of services while the bundle is stopping.
>>>>>>> I think if we remove that bit, the third thread will reject the
>>>>>>> service registration, exit the ManagedServiceFactory#update() and
>>>>>>> release the ManagedServiceFactory lock.
>>>>>>>
>>>>>>> I'll give it a try, but thoughts are welcomed.
>>>>>>>
>>>>>>> --
>>>>>>> ------------------------
>>>>>>> Guillaume Nodet
>>>>>>> ------------------------
>>>>>>> Blog: http://gnodet.blogspot.com/
>>>>>>> ------------------------
>>>>>>> FuseSource, Integration everywhere
>>>>>>> http://fusesource.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ------------------------
>>>>> Guillaume Nodet
>>>>> ------------------------
>>>>> Blog: http://gnodet.blogspot.com/
>>>>> ------------------------
>>>>> FuseSource, Integration everywhere
>>>>> http://fusesource.com
>>>>
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com
>
>
>
> --
> ------------------------
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> FuseSource, Integration everywhere
> http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Guillaume Nodet <gn...@gmail.com>.
... which makes me thing that being able to register services (or at
least try) while the bundle is stopping is still wrong.
What's the use case for that ?
On Thu, May 24, 2012 at 6:05 PM, Guillaume Nodet <gn...@gmail.com> wrote:
> That's actually a really good idea.
> For the record, I came up with the following class
> https://gist.github.com/2782423
> which can't prevent all deadlocks (I think there's still a small
> window of time where the service registration could be attempted while
> the bundle lock is grabbed by the framework when stopping the bundle,
> but before the factory destroy() method is called), but it's way
> better.
> Though, the mere fact I don't see a nice way to avoid the deadlock
> kinda bothers me.
>
> On Thu, May 24, 2012 at 3:58 PM, Marcel Offermans
> <ma...@luminis.nl> wrote:
>> Another pattern you could use here is to have the updated method only validate if the configuration it received is valid and not process it at all in the updated method but delegate that to a different method. That way, you can probably even prevent the creation of the objects you mention in your example altogether. Of course there then has to be "something else" to delegate to (like some other thread).
>>
>> Greetings, Marcel
>>
>>
>> On May 24, 2012, at 15:37 PM, Felix Meschberger wrote:
>>
>>> Hi,
>>>
>>> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>>>
>>>> The assumptions are right.
>>>>
>>>> Do you have any pointer to a well written ManagedServiceFactory that
>>>> can make sure calls to updates are finished before destroying the
>>>> ManagedServiceFactory without any synchronized blocks ?
>>>
>>> Not, off the top of my head.
>>>
>>> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>>>
>>> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>>>
>>> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>>>
>>> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>>>
>>> Regards
>>> Felix
>>>
>>>>
>>>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>>>
>>>>>> I have the following deadlock that sometimes happen:
>>>>>>
>>>>>> Thread 1:
>>>>>
>>>>>> start the bundle
>>>>>> register a ManagedServiceFactory
>>>>>
>>>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>>>
>>>>>>
>>>>>> Thread 2:
>>>>>> stop the bundle
>>>>>
>>>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>>>
>>>>>> grab the bundle lock
>>>>>> try to destroy the ManagedServiceFactory
>>>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>>>
>>>>>> Thread 3:
>>>>>
>>>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>>>
>>>>>> in a different thread, the ConfigAdmin will call the
>>>>>> ManagedServiceFactory#update()
>>>>>> enter synchronized block in the ManagedServiceFactory
>>>>>> register a service
>>>>>> try to grab the bundle lock
>>>>>>
>>>>>>
>>>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>>>> has to be synchronized in order for the destruction of the
>>>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>>>> created services.
>>>>>
>>>>> I disagree.
>>>>>
>>>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>>>
>>>>> Regards
>>>>> Felix
>>>>>
>>>>>>
>>>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>>>> registration of services while the bundle is stopping.
>>>>>> I think if we remove that bit, the third thread will reject the
>>>>>> service registration, exit the ManagedServiceFactory#update() and
>>>>>> release the ManagedServiceFactory lock.
>>>>>>
>>>>>> I'll give it a try, but thoughts are welcomed.
>>>>>>
>>>>>> --
>>>>>> ------------------------
>>>>>> Guillaume Nodet
>>>>>> ------------------------
>>>>>> Blog: http://gnodet.blogspot.com/
>>>>>> ------------------------
>>>>>> FuseSource, Integration everywhere
>>>>>> http://fusesource.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ------------------------
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> FuseSource, Integration everywhere
>>>> http://fusesource.com
>>>
>>>
>>>
>>
>
>
>
> --
> ------------------------
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> FuseSource, Integration everywhere
> http://fusesource.com
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Guillaume Nodet <gn...@gmail.com>.
That's actually a really good idea.
For the record, I came up with the following class
https://gist.github.com/2782423
which can't prevent all deadlocks (I think there's still a small
window of time where the service registration could be attempted while
the bundle lock is grabbed by the framework when stopping the bundle,
but before the factory destroy() method is called), but it's way
better.
Though, the mere fact I don't see a nice way to avoid the deadlock
kinda bothers me.
On Thu, May 24, 2012 at 3:58 PM, Marcel Offermans
<ma...@luminis.nl> wrote:
> Another pattern you could use here is to have the updated method only validate if the configuration it received is valid and not process it at all in the updated method but delegate that to a different method. That way, you can probably even prevent the creation of the objects you mention in your example altogether. Of course there then has to be "something else" to delegate to (like some other thread).
>
> Greetings, Marcel
>
>
> On May 24, 2012, at 15:37 PM, Felix Meschberger wrote:
>
>> Hi,
>>
>> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>>
>>> The assumptions are right.
>>>
>>> Do you have any pointer to a well written ManagedServiceFactory that
>>> can make sure calls to updates are finished before destroying the
>>> ManagedServiceFactory without any synchronized blocks ?
>>
>> Not, off the top of my head.
>>
>> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>>
>> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>>
>> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>>
>> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>>
>> Regards
>> Felix
>>
>>>
>>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>>> Hi,
>>>>
>>>>
>>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>>
>>>>> I have the following deadlock that sometimes happen:
>>>>>
>>>>> Thread 1:
>>>>
>>>>> start the bundle
>>>>> register a ManagedServiceFactory
>>>>
>>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>>
>>>>>
>>>>> Thread 2:
>>>>> stop the bundle
>>>>
>>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>>
>>>>> grab the bundle lock
>>>>> try to destroy the ManagedServiceFactory
>>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>>
>>>>> Thread 3:
>>>>
>>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>>
>>>>> in a different thread, the ConfigAdmin will call the
>>>>> ManagedServiceFactory#update()
>>>>> enter synchronized block in the ManagedServiceFactory
>>>>> register a service
>>>>> try to grab the bundle lock
>>>>>
>>>>>
>>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>>> has to be synchronized in order for the destruction of the
>>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>>> created services.
>>>>
>>>> I disagree.
>>>>
>>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>>
>>>> Regards
>>>> Felix
>>>>
>>>>>
>>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>>> registration of services while the bundle is stopping.
>>>>> I think if we remove that bit, the third thread will reject the
>>>>> service registration, exit the ManagedServiceFactory#update() and
>>>>> release the ManagedServiceFactory lock.
>>>>>
>>>>> I'll give it a try, but thoughts are welcomed.
>>>>>
>>>>> --
>>>>> ------------------------
>>>>> Guillaume Nodet
>>>>> ------------------------
>>>>> Blog: http://gnodet.blogspot.com/
>>>>> ------------------------
>>>>> FuseSource, Integration everywhere
>>>>> http://fusesource.com
>>>>
>>>
>>>
>>>
>>> --
>>> ------------------------
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>> ------------------------
>>> FuseSource, Integration everywhere
>>> http://fusesource.com
>>
>>
>>
>
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Marcel Offermans <ma...@luminis.nl>.
Another pattern you could use here is to have the updated method only validate if the configuration it received is valid and not process it at all in the updated method but delegate that to a different method. That way, you can probably even prevent the creation of the objects you mention in your example altogether. Of course there then has to be "something else" to delegate to (like some other thread).
Greetings, Marcel
On May 24, 2012, at 15:37 PM, Felix Meschberger wrote:
> Hi,
>
> Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
>
>> The assumptions are right.
>>
>> Do you have any pointer to a well written ManagedServiceFactory that
>> can make sure calls to updates are finished before destroying the
>> ManagedServiceFactory without any synchronized blocks ?
>
> Not, off the top of my head.
>
> I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
>
> I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
>
> For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
>
> Usually, I do work with such a "stopped" flag preventing further concurrent operation.
>
> Regards
> Felix
>
>>
>> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>>> Hi,
>>>
>>>
>>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>>
>>>> I have the following deadlock that sometimes happen:
>>>>
>>>> Thread 1:
>>>
>>>> start the bundle
>>>> register a ManagedServiceFactory
>>>
>>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>>
>>>>
>>>> Thread 2:
>>>> stop the bundle
>>>
>>> Assumption: This followin code runs in the BundleActivator.stop method.
>>>
>>>> grab the bundle lock
>>>> try to destroy the ManagedServiceFactory
>>>> deadlock on grabbing the ManagedServiceFactory lock
>>>>
>>>> Thread 3:
>>>
>>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>>
>>>> in a different thread, the ConfigAdmin will call the
>>>> ManagedServiceFactory#update()
>>>> enter synchronized block in the ManagedServiceFactory
>>>> register a service
>>>> try to grab the bundle lock
>>>>
>>>>
>>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>>> has to be synchronized in order for the destruction of the
>>>> ManagedServiceFactory to make sure we destroy all the previously
>>>> created services.
>>>
>>> I disagree.
>>>
>>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>>
>>> Regards
>>> Felix
>>>
>>>>
>>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>>> registration of services while the bundle is stopping.
>>>> I think if we remove that bit, the third thread will reject the
>>>> service registration, exit the ManagedServiceFactory#update() and
>>>> release the ManagedServiceFactory lock.
>>>>
>>>> I'll give it a try, but thoughts are welcomed.
>>>>
>>>> --
>>>> ------------------------
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> FuseSource, Integration everywhere
>>>> http://fusesource.com
>>>
>>
>>
>>
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com
>
>
>
Re: Deadlock with ManagedServiceFactory
Posted by Felix Meschberger <fm...@adobe.com>.
Hi,
Am 24.05.2012 um 15:29 schrieb Guillaume Nodet:
> The assumptions are right.
>
> Do you have any pointer to a well written ManagedServiceFactory that
> can make sure calls to updates are finished before destroying the
> ManagedServiceFactory without any synchronized blocks ?
Not, off the top of my head.
I think it is not possible to fully synchronized (ok you could Java 5 locks which may time out, but after a timeout there is no guarantee, either).
I think the best thing that can be done, is that shutting down a ManagedServiceFactory must just be made lenient.
For example, consider a MSF.updated method creates some object and adds it to a table in the MSF class. When stopping the MSF you start by setting a "stopped" flag and then cleanup all objects in the table. In the updated method you check the stopped flag on entry and only add the generated object to the table at the very end of updated after an additional check for the stopped flag. If the flag is set at that point in time, cleanup the object just created instead of adding it to the table.
Usually, I do work with such a "stopped" flag preventing further concurrent operation.
Regards
Felix
>
> On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
>> Hi,
>>
>>
>> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>>
>>> I have the following deadlock that sometimes happen:
>>>
>>> Thread 1:
>>
>>> start the bundle
>>> register a ManagedServiceFactory
>>
>> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>>
>>>
>>> Thread 2:
>>> stop the bundle
>>
>> Assumption: This followin code runs in the BundleActivator.stop method.
>>
>>> grab the bundle lock
>>> try to destroy the ManagedServiceFactory
>>> deadlock on grabbing the ManagedServiceFactory lock
>>>
>>> Thread 3:
>>
>> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>>
>>> in a different thread, the ConfigAdmin will call the
>>> ManagedServiceFactory#update()
>>> enter synchronized block in the ManagedServiceFactory
>>> register a service
>>> try to grab the bundle lock
>>>
>>>
>>> I don't think the problem comes from my ManagedServiceFactory, as it
>>> has to be synchronized in order for the destruction of the
>>> ManagedServiceFactory to make sure we destroy all the previously
>>> created services.
>>
>> I disagree.
>>
>> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>>
>> Regards
>> Felix
>>
>>>
>>> It seems to me that the problem comes from FELIX-3082 which allows the
>>> registration of services while the bundle is stopping.
>>> I think if we remove that bit, the third thread will reject the
>>> service registration, exit the ManagedServiceFactory#update() and
>>> release the ManagedServiceFactory lock.
>>>
>>> I'll give it a try, but thoughts are welcomed.
>>>
>>> --
>>> ------------------------
>>> Guillaume Nodet
>>> ------------------------
>>> Blog: http://gnodet.blogspot.com/
>>> ------------------------
>>> FuseSource, Integration everywhere
>>> http://fusesource.com
>>
>
>
>
> --
> ------------------------
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> FuseSource, Integration everywhere
> http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Guillaume Nodet <gn...@gmail.com>.
The assumptions are right.
Do you have any pointer to a well written ManagedServiceFactory that
can make sure calls to updates are finished before destroying the
ManagedServiceFactory without any synchronized blocks ?
On Thu, May 24, 2012 at 3:20 PM, Felix Meschberger <fm...@adobe.com> wrote:
> Hi,
>
>
> Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
>
>> I have the following deadlock that sometimes happen:
>>
>> Thread 1:
>
>> start the bundle
>> register a ManagedServiceFactory
>
> Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>
>>
>> Thread 2:
>> stop the bundle
>
> Assumption: This followin code runs in the BundleActivator.stop method.
>
>> grab the bundle lock
>> try to destroy the ManagedServiceFactory
>> deadlock on grabbing the ManagedServiceFactory lock
>>
>> Thread 3:
>
> Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
>
>> in a different thread, the ConfigAdmin will call the
>> ManagedServiceFactory#update()
>> enter synchronized block in the ManagedServiceFactory
>> register a service
>> try to grab the bundle lock
>>
>>
>> I don't think the problem comes from my ManagedServiceFactory, as it
>> has to be synchronized in order for the destruction of the
>> ManagedServiceFactory to make sure we destroy all the previously
>> created services.
>
> I disagree.
>
> I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
>
> Regards
> Felix
>
>>
>> It seems to me that the problem comes from FELIX-3082 which allows the
>> registration of services while the bundle is stopping.
>> I think if we remove that bit, the third thread will reject the
>> service registration, exit the ManagedServiceFactory#update() and
>> release the ManagedServiceFactory lock.
>>
>> I'll give it a try, but thoughts are welcomed.
>>
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com
>
--
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
FuseSource, Integration everywhere
http://fusesource.com
Re: Deadlock with ManagedServiceFactory
Posted by Felix Meschberger <fm...@adobe.com>.
Hi,
Am 24.05.2012 um 12:09 schrieb Guillaume Nodet:
> I have the following deadlock that sometimes happen:
>
> Thread 1:
> start the bundle
> register a ManagedServiceFactory
Assumption: Thread 1 has finished processing when Thread 2 and 3 start with their processing
>
> Thread 2:
> stop the bundle
Assumption: This followin code runs in the BundleActivator.stop method.
> grab the bundle lock
> try to destroy the ManagedServiceFactory
> deadlock on grabbing the ManagedServiceFactory lock
>
> Thread 3:
Assumption: This is the CM_Update thread calling back due to Thread 1's service registration.
> in a different thread, the ConfigAdmin will call the
> ManagedServiceFactory#update()
> enter synchronized block in the ManagedServiceFactory
> register a service
> try to grab the bundle lock
>
>
> I don't think the problem comes from my ManagedServiceFactory, as it
> has to be synchronized in order for the destruction of the
> ManagedServiceFactory to make sure we destroy all the previously
> created services.
I disagree.
I think you are violating the recommendations in section 4.7.3, Synchronization Pitfalls, in the Core Spec.
Regards
Felix
>
> It seems to me that the problem comes from FELIX-3082 which allows the
> registration of services while the bundle is stopping.
> I think if we remove that bit, the third thread will reject the
> service registration, exit the ManagedServiceFactory#update() and
> release the ManagedServiceFactory lock.
>
> I'll give it a try, but thoughts are welcomed.
>
> --
> ------------------------
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> FuseSource, Integration everywhere
> http://fusesource.com