Re: Service isn't initialized on cluster re-activation

[Denis Mekhanikov <dm...@gmail.com>]

Nick,

I agree, that performing heavy work in an exchange thread is generally a
pretty bad idea.
On the other hand, service cancellation in my opinion should happen, when
cluster is in a valid state, when cache operations are still possible.
This way service will be able to store useful information to some cache
before going offline.
I checked, and it doesn't work, when services are cancelled during
deactivation. All operations are getting stuck.
I think, this is the thing, that should be fixed.
If a node is stopped without deactivation, then everything is fine.

If we do cancellation asynchronously, then *cancel() *method won't have a
chance to do anything before node is kicked out of the cluster.
So, user is responsible for making cancel methods deadlock-proof to avoid
problems in the cluster.

What do you think?

Denis

пт, 27 апр. 2018 г. в 8:14, Stanislav Lukyanov <st...@gmail.com>:

> Hi,
>
>
>
> Regarding service redeployment on reactivation – that’s
> https://issues.apache.org/jira/browse/IGNITE-8205,
>
> already fixed in the master branch, will be in 2.6.
>
>
>
> Regarding executing cancel() in the exchange thread – agree, that doesn’t
> seem right.
>
> I guess currently we just run init() and cancel() in whichever thread
> causes them to be called
>
> (I wonder if fix the for IGNITE-8205 causes init() to be executed in the
> exchange thread as well – needs to be checked).
>
> The deployment executor of the service processor seems to be a good fit.
>
>
>
> Perhaps the latter can be checked and fixed in the IEP-17 (
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-17:+Oil+Change+in+Service+Grid
> ).
>
> **Denis Mekhanikov**, could you please tell if execution of service
> methods in possibly inappropriate threads
>
> is on the radar of the IEP-17 already, or does it make sense to add a new
> issue in the list?
>
>
>
> Thanks,
>
> Stan
>
>
>
> *From: *npordash <ni...@gmail.com>
> *Sent: *27 апреля 2018 г. 3:05
> *To: *user@ignite.apache.org
> *Subject: *Service isn't initialized on cluster re-activation
>
>
>
> Hi,
>
>
>
> I was doing some testing where I had some services deployed, then
>
> deactivated the cluster, then re-activated the cluster. I was expecting
> that
>
> the services would have their init and execute methods called again, but
>
> this did not happen. If I proceed to shutdown a node then the cancel method
>
> is called even though the service never went through proper initialization.
>
>
>
> I'm observing this behavior regardless if persistence is enabled or not;
>
> however, if persistence is enabled and I completely restart the cluster
> then
>
> all services are re-initialized as expected.
>
>
>
> This is on Ignite 2.4 and a quick reproducer is:
>
>
>
>
>
> I see the following output when run:
>
>
>
>
>
> In addition, you'll also notice that when the service is cancelled because
>
> of cluster deactivation that this occurs from the exchange thread which
>
> seems incredibly dangerous to me since it can cause the cluster to deadlock
>
> if you do anything in the cancel method that may block (f.e. any call that
>
> leads to GridClusterStateProcessor.publicApiActiveState(true)).
>
>
>
> I found IGNITE-8279 but upfront this appears to be a different issue, but
>
> not sure if it's related to what I'm observing as this use-case isn't using
>
> a client.
>
>
>
> -Nick
>
>
>
>
>
>
>
> --
>
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>

Re: Service isn't initialized on cluster re-activation

[npordash <ni...@gmail.com>]

Got it, makes sense. Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Service isn't initialized on cluster re-activation

[Denis Mekhanikov <dm...@gmail.com>]

Nick,

*TcpDiscoveryMetricsUpdateMessage* (aka heartbeats) are processed in the
discovery thread, but not the exchange.
So, if your *cancel() *method is hanging, then an exchange won't be able to
proceed, but discovery will work fine.
Hanging deactivation is not a pleasant thing, but I think, this is a fair
consequence of a badly implemented *cancel()* method.

Denis

вт, 15 мая 2018 г. в 20:24, npordash <ni...@gmail.com>:

> Hey Denis,
>
> I agree with your assessment. I think it's unexpected that the cluster
> transitions to inactive state prior to services being given a chance to
> shutdown gracefully.
>
> However, to protect the overall cluster health I think we do need some kind
> of protection against errant services with cancel methods that may block
> for
> an unacceptable period of time. What we probably don't want to have happen
> is that the exchange thread blocks for so long while trying to shutdown
> services that it can't handle heartbeats within the configured failure
> detection timeout and trigger a node failure.
>
> Perhaps during deactivation the cancel method can be invoked in the service
> deployment thread pool and the exchange thread can wait for that take to
> complete until some timeout is reached before it gives up. Between each
> service cancellation it would need to prioritize heartbeat handling to
> prevent node failures.
>
> WDYT?
>
> -Nick
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Service isn't initialized on cluster re-activation

[npordash <ni...@gmail.com>]

Hey Denis,

I agree with your assessment. I think it's unexpected that the cluster
transitions to inactive state prior to services being given a chance to
shutdown gracefully.

However, to protect the overall cluster health I think we do need some kind
of protection against errant services with cancel methods that may block for
an unacceptable period of time. What we probably don't want to have happen
is that the exchange thread blocks for so long while trying to shutdown
services that it can't handle heartbeats within the configured failure
detection timeout and trigger a node failure.

Perhaps during deactivation the cancel method can be invoked in the service
deployment thread pool and the exchange thread can wait for that take to
complete until some timeout is reached before it gives up. Between each
service cancellation it would need to prioritize heartbeat handling to
prevent node failures.

WDYT?

-Nick



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/