Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

I've been burrowing into the lifecycle code recently and posted 
https://issues.apache.org/jira/browse/FLUME-1257 regarding it(more 
specifically, regarding inability to shutdown after a component gets 
stuck in a starting loop).

Right now, to put it bluntly, the shutdown model feels incorrect, there 
are a lot of places with a loop waiting for changes in state, that have 
an exception handler for InterruptedException, but nothing will ever 
trigger that interrupt. SIGINT is handled by the shutdown hook, which 
requests a clean shutdown. If a clean shutdown doesn't succeed, 
everything just hangs in limbo until you forcibly kill it.

I'd like to start some kind of discussion on how to go about handling 
this.  One  argument may be that it's just not a problem... so long as 
everything starts and stops in a timely manner. Other simple kludges 
would be setting a limit to number of times through each loop. Or we 
could just start sending interrupts after a  predetermined delay.

Another possibility is adding more lifecycle states to better 
communicate state(starting/stopping?) so that actions can be  aborted as 
necessary. This would likely require all existing components to 
communicate these new states.

Feedback/opinions would be appreciated

Re: Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

On 06/08/2012 11:25 AM, Eric Sammer wrote:
> On Thu, Jun 7, 2012 at 7:12 PM, Juhani Connolly<
> juhani_connolly@cyberagent.co.jp>  wrote:
>
>> Thanks for your input on this.
>>
>> I'm more than happy for misbehaving things to just fail, though I hope
>> that we can leave proper logs so people can know something is wrong.
>>
> Completely agree. +1.
>
>
>> Do you feel the auto restarting behaviour should still remain a part of
>> the lifecycle? I'm not entirely sure if it is something the supervisor
>> should be doing, and the majority of components that fail tend to just keep
>> failing.
>>
> I do like the idea of erlang's always restart policy way of doing things.
> The rationale is that if something is failing  due to an outside resource
> (let's say the file channel due to a lack of disk space) correcting the
> external component should effect recovery of flume. In other words, if a
> disk files, I want to delete some files and let flume get back to work
> without needing to figure out if it's still within its retry period. I say
> let it go forever. Maybe we don't log at some point, but I'm inclined to
> just let it go. If users don't want the logs, they can configure log4j
> conservatively and use metrics to track the number of component failures /
> restarts (which should be exposed, IMO).
>
Ok, I'm not familiar with Erlang and its framework, but sounds good to me.

>> With these simplicity needs in mind though, the starting point that Brock
>> has attached to the JIRA looks simple and performs most  of what we need.
>> I've been working on something bigger than that, but with more settings for
>> policy(configurable policy restricting number of attempts to start/stop,
>> timeouts, and forced termination). This probably isn't necessary though if
>> we decide on some "correct" behavior for components and enforce it in code
>> reviews.
>
> My general feeling is that the framework (flume, proper) should be dumb;
> restart forever or never, configurable by the user by way of a policy
> (which we don't expose today, admittedly). Sources and sinks should not
> enforce lifecycle decisions; a source should do what it's told, as should a
> sink. Weird magic numbers don't buy you much (e.g. "restart three times and
> then give up").
>
> The one thing I didn't implement from erlang in the supervisor code was
> restart policies that impact other components owned by the same supervisor.
> Ex: erlang's supervisors permit one to say "if you restart component X for
> some reason, also restart Y, Z, and A." This is nice as it allows one to
> discard state associated with a previous incarnation of the lifecycle. I
> didn't have a strong need for it, though. In fact, my gut tells me the
> *best* thing to do is go one step further than discard the instances of the
> components and use the last configuration event and builders to
> re-instantiate the components to prevent state leak after an exception that
> may have left a component in a bad state. That's more complicated though.
>
> Sorry, these are just ideas that have been rolling around that I never got
> to really put out there on the lists. I'm brain-vomitting at you. :)

These ideas sound cool, but not exactly easy  to get straight away. 
Hopefully I can start with something more simple and we can iterate from 
there without adding more complexity than is absolutely necessary. I 
plan to put in a rudimentary policy with hooks to inform it of stuff, 
and we can if necessary isolate complex logic in there at a later date.

>
>> On 06/07/2012 03:32 PM, Eric Sammer wrote:
>>
>>> I can try and answer any lingering questions about the existing lifecycle
>>> management code. I knew there were outstanding issues in it (which was the
>>> impetus for that JIRA to move to Guava's service model) but I just never
>>> was able to put in the time. I'm in favor of moving to the Guava
>>> implementation.
>>>
>>> More generally, I strongly believe in well defined semantics and simple
>>> contracts. In other words, we should not get into the business of
>>> attempting to deal with byzantine failure. The idea is that the shutdown
>>> handler (in response to SIGINT) should request an orderly shutdown. If the
>>> system is in good working order, it should do so. If, for instance,
>>> something PermGen OOMs or there's incorrect behavior in a LifecycleAware
>>> component (i.e. a component that does not respect the contract), we should
>>> explicitly *not* try and handle that and a forced kill is proper. My vote
>>> is just to avoid insanely complex logic to deal with incorrectly
>>> implemented components; that always leads to insanely complicated code
>>> that
>>> doesn't always work when things are correctly implemented. Not to mention,
>>> process stop events suffer from the halting problem[1] anyway...
>>>
>>> [1] http://bit.ly/Kz5GMJ
>>>
>>> Thanks for taking this on guys. It's not sexy work, but it's super
>>> important.
>>>
>>> On Wed, Jun 6, 2012 at 8:01 PM, Juhani Connolly<
>>> juhani_connolly@cyberagent.co.**jp<ju...@cyberagent.co.jp>>
>>>   wrote:
>>>
>>>   The biggest barrier to this right now is with the restarting behavior our
>>>> current lifecycle model has, which is not part of the guava lifecycle. It
>>>> means if we're to restart services we need to store everything needed to
>>>> build a new service when the old one dies, and start that. In essence
>>>> we're
>>>> going to need an outer layer to watch the inner(guava layer), which sort
>>>> of
>>>> defeats the purpose.
>>>>
>>>> I'm trying to figure out if there's a way to get around this, or if
>>>> switching from a restarting model to guavas starting/running/stopping/**
>>>> **terminated
>>>>
>>>> model is possible(this would probably require some components to take
>>>> better care of themselves as once they fail they wouldn't auto-restart)
>>>>
>>>>
>>>> On 06/06/2012 05:48 PM, Hari Shreedharan wrote:
>>>>
>>>>   Juhani,
>>>>> It would be interesting to see how much of an effort it would be to
>>>>> replace the current system with Guava. It would be nice to see an
>>>>> initial
>>>>> proof of concept, maybe you can post it on the dev list. I think there
>>>>> would be others would also have ideas and feel the need to update the
>>>>> Lifecycle system.
>>>>>
>>>>> Thanks
>>>>> Hari
>>>>>
>>>>>
>>>>>
>

Re: Endless loops and lack of interrupts

[Eric Sammer <es...@cloudera.com>]

On Thu, Jun 7, 2012 at 7:12 PM, Juhani Connolly <
juhani_connolly@cyberagent.co.jp> wrote:

> Thanks for your input on this.
>
> I'm more than happy for misbehaving things to just fail, though I hope
> that we can leave proper logs so people can know something is wrong.
>

Completely agree. +1.


> Do you feel the auto restarting behaviour should still remain a part of
> the lifecycle? I'm not entirely sure if it is something the supervisor
> should be doing, and the majority of components that fail tend to just keep
> failing.
>

I do like the idea of erlang's always restart policy way of doing things.
The rationale is that if something is failing  due to an outside resource
(let's say the file channel due to a lack of disk space) correcting the
external component should effect recovery of flume. In other words, if a
disk files, I want to delete some files and let flume get back to work
without needing to figure out if it's still within its retry period. I say
let it go forever. Maybe we don't log at some point, but I'm inclined to
just let it go. If users don't want the logs, they can configure log4j
conservatively and use metrics to track the number of component failures /
restarts (which should be exposed, IMO).


> With these simplicity needs in mind though, the starting point that Brock
> has attached to the JIRA looks simple and performs most  of what we need.
> I've been working on something bigger than that, but with more settings for
> policy(configurable policy restricting number of attempts to start/stop,
> timeouts, and forced termination). This probably isn't necessary though if
> we decide on some "correct" behavior for components and enforce it in code
> reviews.


My general feeling is that the framework (flume, proper) should be dumb;
restart forever or never, configurable by the user by way of a policy
(which we don't expose today, admittedly). Sources and sinks should not
enforce lifecycle decisions; a source should do what it's told, as should a
sink. Weird magic numbers don't buy you much (e.g. "restart three times and
then give up").

The one thing I didn't implement from erlang in the supervisor code was
restart policies that impact other components owned by the same supervisor.
Ex: erlang's supervisors permit one to say "if you restart component X for
some reason, also restart Y, Z, and A." This is nice as it allows one to
discard state associated with a previous incarnation of the lifecycle. I
didn't have a strong need for it, though. In fact, my gut tells me the
*best* thing to do is go one step further than discard the instances of the
components and use the last configuration event and builders to
re-instantiate the components to prevent state leak after an exception that
may have left a component in a bad state. That's more complicated though.

Sorry, these are just ideas that have been rolling around that I never got
to really put out there on the lists. I'm brain-vomitting at you. :)


>
> On 06/07/2012 03:32 PM, Eric Sammer wrote:
>
>> I can try and answer any lingering questions about the existing lifecycle
>> management code. I knew there were outstanding issues in it (which was the
>> impetus for that JIRA to move to Guava's service model) but I just never
>> was able to put in the time. I'm in favor of moving to the Guava
>> implementation.
>>
>> More generally, I strongly believe in well defined semantics and simple
>> contracts. In other words, we should not get into the business of
>> attempting to deal with byzantine failure. The idea is that the shutdown
>> handler (in response to SIGINT) should request an orderly shutdown. If the
>> system is in good working order, it should do so. If, for instance,
>> something PermGen OOMs or there's incorrect behavior in a LifecycleAware
>> component (i.e. a component that does not respect the contract), we should
>> explicitly *not* try and handle that and a forced kill is proper. My vote
>> is just to avoid insanely complex logic to deal with incorrectly
>> implemented components; that always leads to insanely complicated code
>> that
>> doesn't always work when things are correctly implemented. Not to mention,
>> process stop events suffer from the halting problem[1] anyway...
>>
>> [1] http://bit.ly/Kz5GMJ
>>
>> Thanks for taking this on guys. It's not sexy work, but it's super
>> important.
>>
>> On Wed, Jun 6, 2012 at 8:01 PM, Juhani Connolly<
>> juhani_connolly@cyberagent.co.**jp <ju...@cyberagent.co.jp>>
>>  wrote:
>>
>>  The biggest barrier to this right now is with the restarting behavior our
>>> current lifecycle model has, which is not part of the guava lifecycle. It
>>> means if we're to restart services we need to store everything needed to
>>> build a new service when the old one dies, and start that. In essence
>>> we're
>>> going to need an outer layer to watch the inner(guava layer), which sort
>>> of
>>> defeats the purpose.
>>>
>>> I'm trying to figure out if there's a way to get around this, or if
>>> switching from a restarting model to guavas starting/running/stopping/**
>>> **terminated
>>>
>>> model is possible(this would probably require some components to take
>>> better care of themselves as once they fail they wouldn't auto-restart)
>>>
>>>
>>> On 06/06/2012 05:48 PM, Hari Shreedharan wrote:
>>>
>>>  Juhani,
>>>>
>>>> It would be interesting to see how much of an effort it would be to
>>>> replace the current system with Guava. It would be nice to see an
>>>> initial
>>>> proof of concept, maybe you can post it on the dev list. I think there
>>>> would be others would also have ideas and feel the need to update the
>>>> Lifecycle system.
>>>>
>>>> Thanks
>>>> Hari
>>>>
>>>>
>>>>
>>
>


-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

Re: Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

Thanks for your input on this.

I'm more than happy for misbehaving things to just fail, though I hope 
that we can leave proper logs so people can know something is wrong.

Do you feel the auto restarting behaviour should still remain a part of 
the lifecycle? I'm not entirely sure if it is something the supervisor 
should be doing, and the majority of components that fail tend to just 
keep failing.

With these simplicity needs in mind though, the starting point that 
Brock has attached to the JIRA looks simple and performs most  of what 
we need. I've been working on something bigger than that, but with more 
settings for policy(configurable policy restricting number of attempts 
to start/stop, timeouts, and forced termination). This probably isn't 
necessary though if we decide on some "correct" behavior for components 
and enforce it in code reviews.

On 06/07/2012 03:32 PM, Eric Sammer wrote:
> I can try and answer any lingering questions about the existing lifecycle
> management code. I knew there were outstanding issues in it (which was the
> impetus for that JIRA to move to Guava's service model) but I just never
> was able to put in the time. I'm in favor of moving to the Guava
> implementation.
>
> More generally, I strongly believe in well defined semantics and simple
> contracts. In other words, we should not get into the business of
> attempting to deal with byzantine failure. The idea is that the shutdown
> handler (in response to SIGINT) should request an orderly shutdown. If the
> system is in good working order, it should do so. If, for instance,
> something PermGen OOMs or there's incorrect behavior in a LifecycleAware
> component (i.e. a component that does not respect the contract), we should
> explicitly *not* try and handle that and a forced kill is proper. My vote
> is just to avoid insanely complex logic to deal with incorrectly
> implemented components; that always leads to insanely complicated code that
> doesn't always work when things are correctly implemented. Not to mention,
> process stop events suffer from the halting problem[1] anyway...
>
> [1] http://bit.ly/Kz5GMJ
>
> Thanks for taking this on guys. It's not sexy work, but it's super
> important.
>
> On Wed, Jun 6, 2012 at 8:01 PM, Juhani Connolly<
> juhani_connolly@cyberagent.co.jp>  wrote:
>
>> The biggest barrier to this right now is with the restarting behavior our
>> current lifecycle model has, which is not part of the guava lifecycle. It
>> means if we're to restart services we need to store everything needed to
>> build a new service when the old one dies, and start that. In essence we're
>> going to need an outer layer to watch the inner(guava layer), which sort of
>> defeats the purpose.
>>
>> I'm trying to figure out if there's a way to get around this, or if
>> switching from a restarting model to guavas starting/running/stopping/**terminated
>> model is possible(this would probably require some components to take
>> better care of themselves as once they fail they wouldn't auto-restart)
>>
>>
>> On 06/06/2012 05:48 PM, Hari Shreedharan wrote:
>>
>>> Juhani,
>>>
>>> It would be interesting to see how much of an effort it would be to
>>> replace the current system with Guava. It would be nice to see an initial
>>> proof of concept, maybe you can post it on the dev list. I think there
>>> would be others would also have ideas and feel the need to update the
>>> Lifecycle system.
>>>
>>> Thanks
>>> Hari
>>>
>>>
>

Re: Endless loops and lack of interrupts

[Eric Sammer <es...@cloudera.com>]

I can try and answer any lingering questions about the existing lifecycle
management code. I knew there were outstanding issues in it (which was the
impetus for that JIRA to move to Guava's service model) but I just never
was able to put in the time. I'm in favor of moving to the Guava
implementation.

More generally, I strongly believe in well defined semantics and simple
contracts. In other words, we should not get into the business of
attempting to deal with byzantine failure. The idea is that the shutdown
handler (in response to SIGINT) should request an orderly shutdown. If the
system is in good working order, it should do so. If, for instance,
something PermGen OOMs or there's incorrect behavior in a LifecycleAware
component (i.e. a component that does not respect the contract), we should
explicitly *not* try and handle that and a forced kill is proper. My vote
is just to avoid insanely complex logic to deal with incorrectly
implemented components; that always leads to insanely complicated code that
doesn't always work when things are correctly implemented. Not to mention,
process stop events suffer from the halting problem[1] anyway...

[1] http://bit.ly/Kz5GMJ

Thanks for taking this on guys. It's not sexy work, but it's super
important.

On Wed, Jun 6, 2012 at 8:01 PM, Juhani Connolly <
juhani_connolly@cyberagent.co.jp> wrote:

> The biggest barrier to this right now is with the restarting behavior our
> current lifecycle model has, which is not part of the guava lifecycle. It
> means if we're to restart services we need to store everything needed to
> build a new service when the old one dies, and start that. In essence we're
> going to need an outer layer to watch the inner(guava layer), which sort of
> defeats the purpose.
>
> I'm trying to figure out if there's a way to get around this, or if
> switching from a restarting model to guavas starting/running/stopping/**terminated
> model is possible(this would probably require some components to take
> better care of themselves as once they fail they wouldn't auto-restart)
>
>
> On 06/06/2012 05:48 PM, Hari Shreedharan wrote:
>
>> Juhani,
>>
>> It would be interesting to see how much of an effort it would be to
>> replace the current system with Guava. It would be nice to see an initial
>> proof of concept, maybe you can post it on the dev list. I think there
>> would be others would also have ideas and feel the need to update the
>> Lifecycle system.
>>
>> Thanks
>> Hari
>>
>>
>


-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

Re: Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

The biggest barrier to this right now is with the restarting behavior 
our current lifecycle model has, which is not part of the guava 
lifecycle. It means if we're to restart services we need to store 
everything needed to build a new service when the old one dies, and 
start that. In essence we're going to need an outer layer to watch the 
inner(guava layer), which sort of defeats the purpose.

I'm trying to figure out if there's a way to get around this, or if 
switching from a restarting model to guavas 
starting/running/stopping/terminated model is possible(this would 
probably require some components to take better care of themselves as 
once they fail they wouldn't auto-restart)

On 06/06/2012 05:48 PM, Hari Shreedharan wrote:
> Juhani,
>
> It would be interesting to see how much of an effort it would be to replace the current system with Guava. It would be nice to see an initial proof of concept, maybe you can post it on the dev list. I think there would be others would also have ideas and feel the need to update the Lifecycle system.
>
> Thanks
> Hari
>

Re: Endless loops and lack of interrupts

[Hari Shreedharan <hs...@cloudera.com>]

Juhani, 

It would be interesting to see how much of an effort it would be to replace the current system with Guava. It would be nice to see an initial proof of concept, maybe you can post it on the dev list. I think there would be others would also have ideas and feel the need to update the Lifecycle system.

Thanks
Hari

-- 
Hari Shreedharan


On Wednesday, June 6, 2012 at 1:41 AM, Juhani Connolly wrote:

> I too am willing to work on this, and have a bit of time available at 
> the moment. I'm going to fiddle about with a bit and see if I can get 
> some proof of concept to post to the issue. Please don't let that stop 
> you from doing the same if you're interested, as it would be good to 
> spark some discussion.
> 
> On 06/06/2012 03:26 PM, Hari Shreedharan wrote:
> > I believe the long term plan was to move to Guava, as Jarcec pointed out. This is probably better than trying to fix and improve the current lifecycle management system. As it is, there are a bunch of issues with our current system, which have noticed and fixed. I am not quite satisfied with the fixes we have made though, most are tacky and break-fix.
> > 
> > Also, none of our components really check for interrupts and many(most?) don'g have clean shutdown semantics. Maybe this should be discussed and we should come up with a detailed analysis of the situation. I am willing to look at the migration to Guava if no one else has already looked at it or is looking into it.
> > 
> > Thanks
> > Hari
> > 
> 
> 
> 

Re: Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

I too am willing to work on this, and have a bit of time available at 
the moment. I'm going to fiddle about with a bit and see if I can get 
some proof of concept to post to the issue. Please don't let that stop 
you from doing the same if you're interested, as it would be good to 
spark some discussion.

On 06/06/2012 03:26 PM, Hari Shreedharan wrote:
> I believe the long term plan was to move to Guava, as Jarcec pointed out. This is probably better than trying to fix and improve the current lifecycle management system. As it is, there are a bunch of issues with our current system, which  have noticed and fixed. I am not quite satisfied with the fixes we have made though, most are tacky and break-fix.
>
> Also, none of our components really check for interrupts and many(most?) don'g have clean shutdown semantics. Maybe this should be discussed and we should come up with a detailed analysis of the situation. I am willing to look at the migration to Guava if no one else has already looked at it or is looking into it.
>
> Thanks
> Hari
>

Re: Endless loops and lack of interrupts

[Juhani Connolly <ju...@cyberagent.co.jp>]

I missed the Guava ticket, I'll go have a look at it and link it from 
the new issue I had posted.

And yes, it may well be better to just move across. Sounds like a pretty 
epic patch though.

A discussion of shutdown semantics in general would be good. The whole 
startup/shutdown system is sorely lacking in documenting intended usage 
patterns. There should definitely be some kind of method to the madness 
as far as components failing to start is concerned.

On 06/06/2012 03:26 PM, Hari Shreedharan wrote:
> I believe the long term plan was to move to Guava, as Jarcec pointed out. This is probably better than trying to fix and improve the current lifecycle management system. As it is, there are a bunch of issues with our current system, which  have noticed and fixed. I am not quite satisfied with the fixes we have made though, most are tacky and break-fix.
>
> Also, none of our components really check for interrupts and many(most?) don'g have clean shutdown semantics. Maybe this should be discussed and we should come up with a detailed analysis of the situation. I am willing to look at the migration to Guava if no one else has already looked at it or is looking into it.
>
> Thanks
> Hari
>

Re: Endless loops and lack of interrupts

[Hari Shreedharan <hs...@cloudera.com>]

I believe the long term plan was to move to Guava, as Jarcec pointed out. This is probably better than trying to fix and improve the current lifecycle management system. As it is, there are a bunch of issues with our current system, which  have noticed and fixed. I am not quite satisfied with the fixes we have made though, most are tacky and break-fix.  

Also, none of our components really check for interrupts and many(most?) don'g have clean shutdown semantics. Maybe this should be discussed and we should come up with a detailed analysis of the situation. I am willing to look at the migration to Guava if no one else has already looked at it or is looking into it.

Thanks
Hari

-- 
Hari Shreedharan


On Tuesday, June 5, 2012 at 11:13 PM, Jarek Jarcec Cecho wrote:

> Just a quick question to the idea of adding more lifecycle states - are we still planning to move from our "homemade" lifecycle management to Guava as is describe in FLUME-966?
> 
> Jarcec
> 
> On Jun 6, 2012, at 8:06 AM, Juhani Connolly wrote:
> 
> > I've been burrowing into the lifecycle code recently and posted https://issues.apache.org/jira/browse/FLUME-1257 regarding it(more specifically, regarding inability to shutdown after a component gets stuck in a starting loop).
> > 
> > Right now, to put it bluntly, the shutdown model feels incorrect, there are a lot of places with a loop waiting for changes in state, that have an exception handler for InterruptedException, but nothing will ever trigger that interrupt. SIGINT is handled by the shutdown hook, which requests a clean shutdown. If a clean shutdown doesn't succeed, everything just hangs in limbo until you forcibly kill it.
> > 
> > I'd like to start some kind of discussion on how to go about handling this. One argument may be that it's just not a problem... so long as everything starts and stops in a timely manner. Other simple kludges would be setting a limit to number of times through each loop. Or we could just start sending interrupts after a predetermined delay.
> > 
> > Another possibility is adding more lifecycle states to better communicate state(starting/stopping?) so that actions can be aborted as necessary. This would likely require all existing components to communicate these new states.
> > 
> > Feedback/opinions would be appreciated 

Re: Endless loops and lack of interrupts

[Jarek Jarcec Cecho <ja...@apache.org>]

Just a quick question to the idea of adding more lifecycle states - are we still planning to move from our "homemade" lifecycle management to Guava as is describe in FLUME-966?

Jarcec

On Jun 6, 2012, at 8:06 AM, Juhani Connolly wrote:

> I've been burrowing into the lifecycle code recently and posted https://issues.apache.org/jira/browse/FLUME-1257 regarding it(more specifically, regarding inability to shutdown after a component gets stuck in a starting loop).
> 
> Right now, to put it bluntly, the shutdown model feels incorrect, there are a lot of places with a loop waiting for changes in state, that have an exception handler for InterruptedException, but nothing will ever trigger that interrupt. SIGINT is handled by the shutdown hook, which requests a clean shutdown. If a clean shutdown doesn't succeed, everything just hangs in limbo until you forcibly kill it.
> 
> I'd like to start some kind of discussion on how to go about handling this.  One  argument may be that it's just not a problem... so long as everything starts and stops in a timely manner. Other simple kludges would be setting a limit to number of times through each loop. Or we could just start sending interrupts after a  predetermined delay.
> 
> Another possibility is adding more lifecycle states to better communicate state(starting/stopping?) so that actions can be  aborted as necessary. This would likely require all existing components to communicate these new states.
> 
> Feedback/opinions would be appreciated