You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexander Fedotov <al...@gmail.com> on 2017/10/18 15:56:22 UTC
Fix IgniteSpringBean in 2.x
Hi Igniters,
There is an issue that IgniteSpringBean deadlocks when there is CacheStore,
Service, you name it with @SpringResource annotated fields
https://issues.apache.org/jira/browse/IGNITE-6555. The issue appeared after
moving the logic for starting statically configured caches to operate in
the same way as for dynamic ones: from GridDhtPartitionsExchangeFuture.
Deadlock occurs because IgniteSpringBean having acquired internal Spring
lock waits for a GridDhtPartitionsExchangeFuture which is executed in a
separate thread and which in turn tries to inject @SpringResource annotated
fields using the passed Spring application context what leads to an attempt
to acquire the same internal Spring lock that is already being held by the
main thread.
A possible solution is to remove IgniteSpringBean#afterPropertiesSet, make
IgniteSpringBean implement SmartInitializingSingleton and start Ignite
instance inside afterSingletonsInstantiated.
The only possible problem here is that an Ignite bean cannot be
referenced from init-like methods: afterPropertiesSet, @PostConstruct etc.
Any thoughts? Suggestions?
Kind regards,
Alex.
Re: Fix IgniteSpringBean in 2.x
Posted by Alexander Fedotov <al...@gmail.com>.
Val,
When a bean is requested by type/class, which is how SpringResource
injection is done, Spring walks through all *bean definitions* and tries to
figure out a suitable bean by calling to getSingleton method and thus trying
to acquire exactly the same lock. So, any bean could be the reason of this
deadlock. That's why I think that's the best we can do to solve the problem
without drastic changes in the code.
Kind regards,
Alex.
On Sat, Oct 21, 2017 at 2:14 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Alex,
>
> I tried to investigate this scenario and it looks it is reproduced only if:
> - Injected bean is a singleton
> - Injected bean is created *after* IgniteSpringBean
>
> Do you know if there is a way to verify that required singleton is already
> created before injecting it, without acquiring the same global lock? If
> yes, then I would do this and just throw an exception if bean is not ready,
> explaining how to fix it (e.g. using 'depends-on' attribute). This way we
> will not change current semantics and will avoid deadlocks.
>
> What do you think?
>
> -Val
>
> On Wed, Oct 18, 2017 at 12:57 PM, Alexander Fedotov <
> alexander.fedotoff@gmail.com> wrote:
>
> > Hi Val,
> >
> > SmartInitializingSingleton#afterSingletonsInstantiated is invoked right
> at
> > the end of the singleton pre-instantiation phase, *with a guarantee that
> > all regular singleton beans have been created already*.
> > This behavior allows avoiding the deadlock case described in the first
> > message.
> > InitializingBean#afterPropertiesSet method is called when a lock on the
> > Spring's internal collection of singleton beans is being held (due to a
> > singleton bean instantiation)
> > and not all singleton beans have been initialized.
> >
> > With SmartInitializingSingleton approach it won't be possible to call
> > Ignite instance methods inside any kind of init methods of other beans.
> > For example, it won't be possible to call any operation on the Ignite
> > instance inside @PostConstruct annotated methods because the Ignite
> > instance won't be started by that time.
> >
> > Kind regards,
> > Alex.
> >
> > On Wed, Oct 18, 2017 at 8:50 PM, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Hi Alex,
> > >
> > > This indeed has to be fixed, I've seen this issue multiple times.
> > >
> > > Can you please clarify the difference between
> SmartInitializingSingleton
> > > and InitializingBean? And what exactly will not work if we switch? Can
> > you
> > > give an example?
> > >
> > > -Val
> > >
> > > On Wed, Oct 18, 2017 at 8:56 AM, Alexander Fedotov <
> > > alexander.fedotoff@gmail.com> wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > > There is an issue that IgniteSpringBean deadlocks when there is
> > > CacheStore,
> > > > Service, you name it with @SpringResource annotated fields
> > > > https://issues.apache.org/jira/browse/IGNITE-6555. The issue
> appeared
> > > > after
> > > > moving the logic for starting statically configured caches to operate
> > in
> > > > the same way as for dynamic ones: from GridDhtPartitionsExchangeFutur
> > e.
> > > > Deadlock occurs because IgniteSpringBean having acquired internal
> > Spring
> > > > lock waits for a GridDhtPartitionsExchangeFuture which is executed
> in
> > a
> > > > separate thread and which in turn tries to inject @SpringResource
> > > annotated
> > > > fields using the passed Spring application context what leads to an
> > > attempt
> > > > to acquire the same internal Spring lock that is already being held
> by
> > > the
> > > > main thread.
> > > >
> > > > A possible solution is to remove IgniteSpringBean#
> afterPropertiesSet,
> > > make
> > > > IgniteSpringBean implement SmartInitializingSingleton and start
> Ignite
> > > > instance inside afterSingletonsInstantiated.
> > > > The only possible problem here is that an Ignite bean cannot be
> > > > referenced from init-like methods: afterPropertiesSet, @PostConstruct
> > > etc.
> > > >
> > > > Any thoughts? Suggestions?
> > > >
> > > > Kind regards,
> > > > Alex.
> > > >
> > >
> >
>
Re: Fix IgniteSpringBean in 2.x
Posted by Valentin Kulichenko <va...@gmail.com>.
Alex,
I tried to investigate this scenario and it looks it is reproduced only if:
- Injected bean is a singleton
- Injected bean is created *after* IgniteSpringBean
Do you know if there is a way to verify that required singleton is already
created before injecting it, without acquiring the same global lock? If
yes, then I would do this and just throw an exception if bean is not ready,
explaining how to fix it (e.g. using 'depends-on' attribute). This way we
will not change current semantics and will avoid deadlocks.
What do you think?
-Val
On Wed, Oct 18, 2017 at 12:57 PM, Alexander Fedotov <
alexander.fedotoff@gmail.com> wrote:
> Hi Val,
>
> SmartInitializingSingleton#afterSingletonsInstantiated is invoked right at
> the end of the singleton pre-instantiation phase, *with a guarantee that
> all regular singleton beans have been created already*.
> This behavior allows avoiding the deadlock case described in the first
> message.
> InitializingBean#afterPropertiesSet method is called when a lock on the
> Spring's internal collection of singleton beans is being held (due to a
> singleton bean instantiation)
> and not all singleton beans have been initialized.
>
> With SmartInitializingSingleton approach it won't be possible to call
> Ignite instance methods inside any kind of init methods of other beans.
> For example, it won't be possible to call any operation on the Ignite
> instance inside @PostConstruct annotated methods because the Ignite
> instance won't be started by that time.
>
> Kind regards,
> Alex.
>
> On Wed, Oct 18, 2017 at 8:50 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > Hi Alex,
> >
> > This indeed has to be fixed, I've seen this issue multiple times.
> >
> > Can you please clarify the difference between SmartInitializingSingleton
> > and InitializingBean? And what exactly will not work if we switch? Can
> you
> > give an example?
> >
> > -Val
> >
> > On Wed, Oct 18, 2017 at 8:56 AM, Alexander Fedotov <
> > alexander.fedotoff@gmail.com> wrote:
> >
> > > Hi Igniters,
> > >
> > > There is an issue that IgniteSpringBean deadlocks when there is
> > CacheStore,
> > > Service, you name it with @SpringResource annotated fields
> > > https://issues.apache.org/jira/browse/IGNITE-6555. The issue appeared
> > > after
> > > moving the logic for starting statically configured caches to operate
> in
> > > the same way as for dynamic ones: from GridDhtPartitionsExchangeFutur
> e.
> > > Deadlock occurs because IgniteSpringBean having acquired internal
> Spring
> > > lock waits for a GridDhtPartitionsExchangeFuture which is executed in
> a
> > > separate thread and which in turn tries to inject @SpringResource
> > annotated
> > > fields using the passed Spring application context what leads to an
> > attempt
> > > to acquire the same internal Spring lock that is already being held by
> > the
> > > main thread.
> > >
> > > A possible solution is to remove IgniteSpringBean#afterPropertiesSet,
> > make
> > > IgniteSpringBean implement SmartInitializingSingleton and start Ignite
> > > instance inside afterSingletonsInstantiated.
> > > The only possible problem here is that an Ignite bean cannot be
> > > referenced from init-like methods: afterPropertiesSet, @PostConstruct
> > etc.
> > >
> > > Any thoughts? Suggestions?
> > >
> > > Kind regards,
> > > Alex.
> > >
> >
>
Re: Fix IgniteSpringBean in 2.x
Posted by Alexander Fedotov <al...@gmail.com>.
Hi Val,
SmartInitializingSingleton#afterSingletonsInstantiated is invoked right at
the end of the singleton pre-instantiation phase, *with a guarantee that
all regular singleton beans have been created already*.
This behavior allows avoiding the deadlock case described in the first
message.
InitializingBean#afterPropertiesSet method is called when a lock on the
Spring's internal collection of singleton beans is being held (due to a
singleton bean instantiation)
and not all singleton beans have been initialized.
With SmartInitializingSingleton approach it won't be possible to call
Ignite instance methods inside any kind of init methods of other beans.
For example, it won't be possible to call any operation on the Ignite
instance inside @PostConstruct annotated methods because the Ignite
instance won't be started by that time.
Kind regards,
Alex.
On Wed, Oct 18, 2017 at 8:50 PM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:
> Hi Alex,
>
> This indeed has to be fixed, I've seen this issue multiple times.
>
> Can you please clarify the difference between SmartInitializingSingleton
> and InitializingBean? And what exactly will not work if we switch? Can you
> give an example?
>
> -Val
>
> On Wed, Oct 18, 2017 at 8:56 AM, Alexander Fedotov <
> alexander.fedotoff@gmail.com> wrote:
>
> > Hi Igniters,
> >
> > There is an issue that IgniteSpringBean deadlocks when there is
> CacheStore,
> > Service, you name it with @SpringResource annotated fields
> > https://issues.apache.org/jira/browse/IGNITE-6555. The issue appeared
> > after
> > moving the logic for starting statically configured caches to operate in
> > the same way as for dynamic ones: from GridDhtPartitionsExchangeFuture.
> > Deadlock occurs because IgniteSpringBean having acquired internal Spring
> > lock waits for a GridDhtPartitionsExchangeFuture which is executed in a
> > separate thread and which in turn tries to inject @SpringResource
> annotated
> > fields using the passed Spring application context what leads to an
> attempt
> > to acquire the same internal Spring lock that is already being held by
> the
> > main thread.
> >
> > A possible solution is to remove IgniteSpringBean#afterPropertiesSet,
> make
> > IgniteSpringBean implement SmartInitializingSingleton and start Ignite
> > instance inside afterSingletonsInstantiated.
> > The only possible problem here is that an Ignite bean cannot be
> > referenced from init-like methods: afterPropertiesSet, @PostConstruct
> etc.
> >
> > Any thoughts? Suggestions?
> >
> > Kind regards,
> > Alex.
> >
>
Re: Fix IgniteSpringBean in 2.x
Posted by Valentin Kulichenko <va...@gmail.com>.
Hi Alex,
This indeed has to be fixed, I've seen this issue multiple times.
Can you please clarify the difference between SmartInitializingSingleton
and InitializingBean? And what exactly will not work if we switch? Can you
give an example?
-Val
On Wed, Oct 18, 2017 at 8:56 AM, Alexander Fedotov <
alexander.fedotoff@gmail.com> wrote:
> Hi Igniters,
>
> There is an issue that IgniteSpringBean deadlocks when there is CacheStore,
> Service, you name it with @SpringResource annotated fields
> https://issues.apache.org/jira/browse/IGNITE-6555. The issue appeared
> after
> moving the logic for starting statically configured caches to operate in
> the same way as for dynamic ones: from GridDhtPartitionsExchangeFuture.
> Deadlock occurs because IgniteSpringBean having acquired internal Spring
> lock waits for a GridDhtPartitionsExchangeFuture which is executed in a
> separate thread and which in turn tries to inject @SpringResource annotated
> fields using the passed Spring application context what leads to an attempt
> to acquire the same internal Spring lock that is already being held by the
> main thread.
>
> A possible solution is to remove IgniteSpringBean#afterPropertiesSet, make
> IgniteSpringBean implement SmartInitializingSingleton and start Ignite
> instance inside afterSingletonsInstantiated.
> The only possible problem here is that an Ignite bean cannot be
> referenced from init-like methods: afterPropertiesSet, @PostConstruct etc.
>
> Any thoughts? Suggestions?
>
> Kind regards,
> Alex.
>