You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Bertrand Delacretaz <bd...@apache.org> on 2019/03/26 13:24:52 UTC
Pausing actions, waiting for human input etc.
Hi,
I'm looking at a way to execute long-running compositions of Actions,
including waiting for human inputs with long timeouts in the range of
several days.
The basic pattern is something like
wsk.suspend(someEvent)
// this line executes only after that someEvent is received
AFAIK that's not currently possible out of the box - but I'd be happy
to be proven wrong!
Is anyone working on something similar, or are there any prototypes
around that go in that direction?
Two initial ideas that come to mind is taking inspiration from the way
the Composer's Parallel combinator saves state [1], or pausing between
invocations of Conductor Actions [2]. I haven't looked at the details
of how those work so far.
-Bertrand
[1] https://github.com/apache/incubator-openwhisk-composer/blob/master/docs/COMBINATORS.md#parallel
[2] https://github.com/apache/incubator-openwhisk/blob/master/docs/conductors.md
Re: Pausing actions, waiting for human input etc.
Posted by David Breitgand <DA...@il.ibm.com>.
Hi Bertrand,
AFAIK, this is not currently possible out of the box, but it will be very
useful.
At IBM Research -- Haifa, we are thinking about similar functionality for
long-running serveless pipelenes over a data lake.
I think a generalization of this will be a "state machine", where you can
suspend on a different events.
Maybe something of the kind should be added to Composer?
Thoughts?
-- david
From: Bertrand Delacretaz <bd...@apache.org>
To: OpenWhisk Dev <de...@openwhisk.apache.org>
Date: 26/03/2019 03:25 PM
Subject: Pausing actions, waiting for human input etc.
Hi,
I'm looking at a way to execute long-running compositions of Actions,
including waiting for human inputs with long timeouts in the range of
several days.
The basic pattern is something like
wsk.suspend(someEvent)
// this line executes only after that someEvent is received
AFAIK that's not currently possible out of the box - but I'd be happy
to be proven wrong!
Is anyone working on something similar, or are there any prototypes
around that go in that direction?
Two initial ideas that come to mind is taking inspiration from the way
the Composer's Parallel combinator saves state [1], or pausing between
invocations of Conductor Actions [2]. I haven't looked at the details
of how those work so far.
-Bertrand
[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dopenwhisk-2Dcomposer_blob_master_docs_COMBINATORS.md-23parallel&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=V_hxovuZ2mbhs9xLcmbhQHjweOY8lKOd0dcgx6t1ZDU&m=eboWdranasWuCzZLRloF0r1lmgag7QpiorH6-36gxhQ&s=OhSDKXD0Zi3qXftuAX8QauCoy73j3a_mRRNP-mXXi5I&e=
[2]
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dopenwhisk_blob_master_docs_conductors.md&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=V_hxovuZ2mbhs9xLcmbhQHjweOY8lKOd0dcgx6t1ZDU&m=eboWdranasWuCzZLRloF0r1lmgag7QpiorH6-36gxhQ&s=FogDwUOWga_AHT_E3dRWFuvVpyqEgaJm97zzSyoMIws&e=
Re: Pausing actions, waiting for human input etc.
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
On Wed, Mar 27, 2019 at 4:10 PM Bertrand Delacretaz
<bd...@apache.org> wrote:
> ...Maybe that service can be a set of OpenWhisk Actions...that would
> avoid having to deploy additional services (outside of states
> persistence) while not having to make core changes...
FYI I have played more concretely with that idea and implemented a
prototype that runs state machines on OpenWhisk using the Amazon
States Language, with continuations, at
https://github.com/bdelacretaz/openwhisk-statebox
Feedback is welcome - I'll be busy with other things next week so
might not reply quickly.
-Bertrand
Re: Pausing actions, waiting for human input etc.
Posted by Matt Sicker <bo...@gmail.com>.
What you're describing sounds a lot like what Jenkins does for user
input steps in a build. A stateful task manager is required to do this
as you've been exploring. For debugging purposes, do you think it may
be simpler to write a dedicated service around this, or would a more
serverless approach be that straightforward? I'm not very clear on the
implementation details myself, so this is sort of spitballing here.
On Wed, 27 Mar 2019 at 10:10, Bertrand Delacretaz
<bd...@apache.org> wrote:
>
> Hi,
>
> Thank you everybody for you replies.
>
> I understand implementing this directly in OpenWhisk is challenging,
> and especially this comment from Olivier:
>
> On Tue, Mar 26, 2019 at 6:12 PM Olivier Tardieu <ta...@us.ibm.com> wrote:
> > ...Alternatively, the dynamic creation of trigger-like and rule-like things
> > can be pushed to an external system outside of OpenWhisk. This
> > addresses issue number 2 and can also help with 1...
>
> makes me think that it's worth looking at how a distinct "state
> machine service" can help with that, rather than making complex
> changes to OpenWhisk itself.
>
> Maybe that service can be a set of OpenWhisk Actions...that would
> avoid having to deploy additional services (outside of states
> persistence) while not having to make core changes. At the cost of
> some performance probably, but if it's about slow human-driven or
> batch processing workflows that might not be a real problem.
>
> I'll think about a minimal prototype that would demonstrate that and
> share it here if I succeed.
>
> -Bertrand
--
Matt Sicker <bo...@gmail.com>
Re: Pausing actions, waiting for human input etc.
Posted by Bertrand Delacretaz <bd...@apache.org>.
Hi,
Thank you everybody for you replies.
I understand implementing this directly in OpenWhisk is challenging,
and especially this comment from Olivier:
On Tue, Mar 26, 2019 at 6:12 PM Olivier Tardieu <ta...@us.ibm.com> wrote:
> ...Alternatively, the dynamic creation of trigger-like and rule-like things
> can be pushed to an external system outside of OpenWhisk. This
> addresses issue number 2 and can also help with 1...
makes me think that it's worth looking at how a distinct "state
machine service" can help with that, rather than making complex
changes to OpenWhisk itself.
Maybe that service can be a set of OpenWhisk Actions...that would
avoid having to deploy additional services (outside of states
persistence) while not having to make core changes. At the cost of
some performance probably, but if it's about slow human-driven or
batch processing workflows that might not be a real problem.
I'll think about a minimal prototype that would demonstrate that and
share it here if I succeed.
-Bertrand
Re: Pausing actions, waiting for human input etc.
Posted by Olivier Tardieu <ta...@us.ibm.com>.
Bertrand,
With suspend/resume (or parallel in composer today) you have to give up
substitution
as Rodric pointed out. While results (intermediate or final) of
long-running compositions
can appear in activation records, finding the corresponding activation
records amounts
to finding a needle in a haystack as OpenWhisk default activation database
implementation
does not permit fancy queries. Lack of substitution also means that quotas
and throttles
are in effect. Concretely, an execution may fail to resume because the
account is over its
quota.
It is possible to implement suspend and resume today by dynamically
creating triggers
and rules. An action could in principle create an alarm trigger and a rule
to reinvoke itself
at a later point in time or invoke the next action in a logical sequence.
The trigger can even
embed state. Of course, building actions that can save their state and
resume execution
from a saved state requires either some no-trivial coding or composer-like
code generation.
There are two main concerns with this approach:
1. garbage collection: the triggers and rules should be deleted after use
(relatively easy) or when
something unexpected happen, eg, timeout (harder).
2. while OpenWhisk is meant to handle lots of activations, I don't think
it is meant
to handle lots of creation and deletion operations. But maybe you don't
need a lot.
Alternatively, the dynamic creation of trigger-like and rule-like things
can be pushed to
an external system outside of OpenWhisk. This addresses issue number 2 and
can also
help with 1 as such a system may have built-in support for some form of
self-cleaning.
Handling external inputs as opposed to sleep/alarm scenarios further
complicates things
as order and concurrency control potentially become an issue. Nothing in
OpenWhisk
itself guarantees that external inputs would be processed in order or that
inputs
would not be processed concurrently. Of course, if user inputs are few and
far between
this may not be a concern for your scenario. In general, there is no
simple solution.
I did experiment with these approaches for composer in the past but I did
not feel
there was enough interest in the community to warrant a serious
implementation.
Olivier
Re: Pausing actions, waiting for human input etc.
Posted by Rodric Rabbah <ro...@gmail.com>.
I'll defer to Olivier Tardieu on this topic but brliefly, with an external
medium for preserving state, e.g., Redis (which you need for parallel
combinators), this is possible... This may have even existed at one point
in the past.
Compositions with OpenWhisk intrinsic support provide for three features
that are unique to our orchestration (these were the subject of a
publication [1]):
1. black box actions can be composed (functions are not modified)
2. no double billing (in space or time, ie the conductor action which
orchestrated the composition is not waiting for actions to finish and
holding on to resources)
3. substitution, meaning compositions themselves are further composable
because they are like other actions
With a combinator for suspend/resume, you give up substitution --- since
the system has to generate an activation record for the parent activation
and cannot wait indefinitely. So suspending the composition terminate the
first execution and a later resume creates a new activation. Whether the
two can be linked through some causality reference is desirable but
orthogonal, I think.
[1] The Serverless Trilemma: Function Composition for Serverless Computing
<http://dl.acm.org/citation.cfm?id=3133855>
http://dl.acm.org/citation.cfm?id=3133855
On Tue, Mar 26, 2019 at 9:25 AM Bertrand Delacretaz <bd...@apache.org>
wrote:
> Hi,
>
> I'm looking at a way to execute long-running compositions of Actions,
> including waiting for human inputs with long timeouts in the range of
> several days.
>
> The basic pattern is something like
>
> wsk.suspend(someEvent)
> // this line executes only after that someEvent is received
>
> AFAIK that's not currently possible out of the box - but I'd be happy
> to be proven wrong!
>
> Is anyone working on something similar, or are there any prototypes
> around that go in that direction?
>
> Two initial ideas that come to mind is taking inspiration from the way
> the Composer's Parallel combinator saves state [1], or pausing between
> invocations of Conductor Actions [2]. I haven't looked at the details
> of how those work so far.
>
> -Bertrand
>
> [1]
> https://github.com/apache/incubator-openwhisk-composer/blob/master/docs/COMBINATORS.md#parallel
> [2]
> https://github.com/apache/incubator-openwhisk/blob/master/docs/conductors.md
>