You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Tyson Norris <tn...@adobe.com.INVALID> on 2017/10/04 23:45:24 UTC
Invoker activation queueing proposal
Hi -
I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
In short: Instead of scheduling activations to an invoker on receipt, do the following:
- execute the activation "immediately" if capacity is available
- provide a single overflow topic for activations that cannot execute “immediately"
- schedule from the overflow topic when capacity is available
(BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
I put a wiki page proposal here: https://cwiki.apache.org/confluence/display/OPENWHISK/Invoker+Activation+Queueing+Change
WDYT?
Thanks
Tyson
Re: Invoker activation queueing proposal
Posted by Rodric Rabbah <ro...@gmail.com>.
For N invokers, the number of queues could be 2N (ready and overflow per invoker); or N+1 (ready per invoker and a global overflow). I used two queues to refer to both scenarios - maybe that was confusing?
The ready queue should have a very low occupancy time, since the consuming invoker has capacity, requests in that queue will be drained fast. I’m not sure if you’re implying this shouldn’t be Kafka and instead something else. If so, we can break that out into a separate discussion.
-r
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
OK, yes, was confused by “could be 2N *or* N+1” queues.
RE: occupancy time in ready queue: yes should be super low; stay Kafka for now, agree thats a separate discussion.
RE: overflow queue - I’m not sure the use in having it “per invoker”, so would plan to make it shared (single overflow queue). i.e. If it's per invoker, the same scheduling problem exists, except now you have a chance to reschedule when overflow queue is processed; but in that case, just schedule it once, only at the time it is landing in the ready queue. This makes it a) simpler (fewer topics) and b) arguably more fair(overflow items waiting longest get priority once capacity is available on any invoker).
> On Oct 4, 2017, at 11:41 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>
> The ready queue is per invoker - it has to be (and basically the current implementation; I’m not convinced you need to change that from Kafka at first: what’s the expected occupancy in the Kafka queue, per invoker, vs any other implementation you have in mind?).
>
> The overflow queue, could be per invoker or not. Today both queues are conflated into a single topic per invoker. Hence, two queues.
>
> -r
>
>> On Oct 4, 2017, at 10:49 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>
>> So "dispatches immediately to an invoker" is currently implemented as a Kafka topic-per-invoker. If we don't change that Kafka transport to something else, I'm not sure what you have in mind to get down to "only 2 queues". In other words the "ready queue" seems problematic to dispatch via a single queue to multiple invokers if based on the current Kafka impl.
>>
>>
>>> On Oct 4, 2017, at 6:29 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>>
>>> What you’re describing is generalized to essentially two queues. The ready queue which dispatches immediately to an invoker with capacity and an overflow queue. Whether the latter is one per invoker or global may not matter much as long as it’s drained by the load balancer and not committed otherwise to the invoker ready queue where it is difficult to reassign to an available invoker later (ala work stealing). There are pathologies one has to guard against and we should consider fairness policies as well. There’s good theory in this space that I can imagine using to model the heuristics.
>>>
>>> -r
>>>
>>>> On Oct 4, 2017, at 8:39 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>>
>>>> Not sure what you mean by two queues - you mean two queues per invoker? Or total?
>>>>
>>>>> On Oct 4, 2017, at 5:14 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>>>>
>>>>> A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
>>>>>
>>>>> -r
>>>>>
>>>>>> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>>>>
>>>>>> Hi -
>>>>>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>>>>>
>>>>>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>>>>> - execute the activation "immediately" if capacity is available
>>>>>> - provide a single overflow topic for activations that cannot execute “immediately"
>>>>>> - schedule from the overflow topic when capacity is available
>>>>>>
>>>>>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>>>>>
>>>>>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>>>>>
>>>>>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOPENWHISK%2FInvoker%2BActivation%2BQueueing%2BChange&data=02%7C01%7C%7C3ff41b0527be4e13398c08d50b8614ee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636427592860490625&sdata=afOAkeXRW6iArXa72IGd6gnhVyrOwXsDxnjfSAQqL18%3D&reserved=0
>>>>>>
>>>>>> WDYT?
>>>>>>
>>>>>> Thanks
>>>>>> Tyson
Re: Invoker activation queueing proposal
Posted by Rodric Rabbah <ro...@gmail.com>.
The ready queue is per invoker - it has to be (and basically the current implementation; I’m not convinced you need to change that from Kafka at first: what’s the expected occupancy in the Kafka queue, per invoker, vs any other implementation you have in mind?).
The overflow queue, could be per invoker or not. Today both queues are conflated into a single topic per invoker. Hence, two queues.
-r
> On Oct 4, 2017, at 10:49 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> So "dispatches immediately to an invoker" is currently implemented as a Kafka topic-per-invoker. If we don't change that Kafka transport to something else, I'm not sure what you have in mind to get down to "only 2 queues". In other words the "ready queue" seems problematic to dispatch via a single queue to multiple invokers if based on the current Kafka impl.
>
>
>> On Oct 4, 2017, at 6:29 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>
>> What you’re describing is generalized to essentially two queues. The ready queue which dispatches immediately to an invoker with capacity and an overflow queue. Whether the latter is one per invoker or global may not matter much as long as it’s drained by the load balancer and not committed otherwise to the invoker ready queue where it is difficult to reassign to an available invoker later (ala work stealing). There are pathologies one has to guard against and we should consider fairness policies as well. There’s good theory in this space that I can imagine using to model the heuristics.
>>
>> -r
>>
>>> On Oct 4, 2017, at 8:39 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>
>>> Not sure what you mean by two queues - you mean two queues per invoker? Or total?
>>>
>>>> On Oct 4, 2017, at 5:14 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>>>
>>>> A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
>>>>
>>>> -r
>>>>
>>>>> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>>>
>>>>> Hi -
>>>>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>>>>
>>>>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>>>> - execute the activation "immediately" if capacity is available
>>>>> - provide a single overflow topic for activations that cannot execute “immediately"
>>>>> - schedule from the overflow topic when capacity is available
>>>>>
>>>>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>>>>
>>>>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>>>>
>>>>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOPENWHISK%2FInvoker%2BActivation%2BQueueing%2BChange&data=02%7C01%7C%7C3ff41b0527be4e13398c08d50b8614ee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636427592860490625&sdata=afOAkeXRW6iArXa72IGd6gnhVyrOwXsDxnjfSAQqL18%3D&reserved=0
>>>>>
>>>>> WDYT?
>>>>>
>>>>> Thanks
>>>>> Tyson
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
So "dispatches immediately to an invoker" is currently implemented as a Kafka topic-per-invoker. If we don't change that Kafka transport to something else, I'm not sure what you have in mind to get down to "only 2 queues". In other words the "ready queue" seems problematic to dispatch via a single queue to multiple invokers if based on the current Kafka impl.
> On Oct 4, 2017, at 6:29 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>
> What you’re describing is generalized to essentially two queues. The ready queue which dispatches immediately to an invoker with capacity and an overflow queue. Whether the latter is one per invoker or global may not matter much as long as it’s drained by the load balancer and not committed otherwise to the invoker ready queue where it is difficult to reassign to an available invoker later (ala work stealing). There are pathologies one has to guard against and we should consider fairness policies as well. There’s good theory in this space that I can imagine using to model the heuristics.
>
> -r
>
>> On Oct 4, 2017, at 8:39 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>
>> Not sure what you mean by two queues - you mean two queues per invoker? Or total?
>>
>>> On Oct 4, 2017, at 5:14 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>>
>>> A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
>>>
>>> -r
>>>
>>>> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>>
>>>> Hi -
>>>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>>>
>>>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>>> - execute the activation "immediately" if capacity is available
>>>> - provide a single overflow topic for activations that cannot execute “immediately"
>>>> - schedule from the overflow topic when capacity is available
>>>>
>>>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>>>
>>>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>>>
>>>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOPENWHISK%2FInvoker%2BActivation%2BQueueing%2BChange&data=02%7C01%7C%7C3ff41b0527be4e13398c08d50b8614ee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636427592860490625&sdata=afOAkeXRW6iArXa72IGd6gnhVyrOwXsDxnjfSAQqL18%3D&reserved=0
>>>>
>>>> WDYT?
>>>>
>>>> Thanks
>>>> Tyson
Re: Invoker activation queueing proposal
Posted by Rodric Rabbah <ro...@gmail.com>.
What you’re describing is generalized to essentially two queues. The ready queue which dispatches immediately to an invoker with capacity and an overflow queue. Whether the latter is one per invoker or global may not matter much as long as it’s drained by the load balancer and not committed otherwise to the invoker ready queue where it is difficult to reassign to an available invoker later (ala work stealing). There are pathologies one has to guard against and we should consider fairness policies as well. There’s good theory in this space that I can imagine using to model the heuristics.
-r
> On Oct 4, 2017, at 8:39 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> Not sure what you mean by two queues - you mean two queues per invoker? Or total?
>
>> On Oct 4, 2017, at 5:14 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>>
>> A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
>>
>> -r
>>
>>> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>
>>> Hi -
>>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>>
>>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>> - execute the activation "immediately" if capacity is available
>>> - provide a single overflow topic for activations that cannot execute “immediately"
>>> - schedule from the overflow topic when capacity is available
>>>
>>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>>
>>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>>
>>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOPENWHISK%2FInvoker%2BActivation%2BQueueing%2BChange&data=02%7C01%7C%7C3ff41b0527be4e13398c08d50b8614ee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636427592860490625&sdata=afOAkeXRW6iArXa72IGd6gnhVyrOwXsDxnjfSAQqL18%3D&reserved=0
>>>
>>> WDYT?
>>>
>>> Thanks
>>> Tyson
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
Not sure what you mean by two queues - you mean two queues per invoker? Or total?
> On Oct 4, 2017, at 5:14 PM, Rodric Rabbah <ro...@gmail.com> wrote:
>
> A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
>
> -r
>
>> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>
>> Hi -
>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>
>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>> - execute the activation "immediately" if capacity is available
>> - provide a single overflow topic for activations that cannot execute “immediately"
>> - schedule from the overflow topic when capacity is available
>>
>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>
>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>
>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FOPENWHISK%2FInvoker%2BActivation%2BQueueing%2BChange&data=02%7C01%7C%7C3ff41b0527be4e13398c08d50b8614ee%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636427592860490625&sdata=afOAkeXRW6iArXa72IGd6gnhVyrOwXsDxnjfSAQqL18%3D&reserved=0
>>
>> WDYT?
>>
>> Thanks
>> Tyson
Re: Invoker activation queueing proposal
Posted by Rodric Rabbah <ro...@gmail.com>.
A two queue (topic) approach can mitigate the lack of random access from a Kafka topic/queue once a request is committed (in today’s architecture). This could enable work stealing in particular since the (overflow) queue can be drained upstream (ie load balancer) and reassigned to free invokers. Balancing cold start that container locality would them be a heuristic we can apply more judiciously as capacity is available (versus the current approach which binds too early and prevents rebalancing).
-r
> On Oct 4, 2017, at 7:45 PM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> Hi -
> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>
> In short: Instead of scheduling activations to an invoker on receipt, do the following:
> - execute the activation "immediately" if capacity is available
> - provide a single overflow topic for activations that cannot execute “immediately"
> - schedule from the overflow topic when capacity is available
>
> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>
> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>
> I put a wiki page proposal here: https://cwiki.apache.org/confluence/display/OPENWHISK/Invoker+Activation+Queueing+Change
>
> WDYT?
>
> Thanks
> Tyson
Re: Invoker activation queueing proposal
Posted by Markus Thömmes <ma...@me.com>.
The overflow queue is not intended to be drained by a large scale of invokers but by a relatively low scale of loadbalancers! I don‘t think rebalancing is a problem then.
That loadbalancer basically has 2 states:
1. capacity is there: schedule everything you get and put it on immediate invoker queues.
2. overflow: put everything you get on the overflow queue and only schedule stuff that you drain from that queue to the immediate queues.
Overflow state is over once the overflow queue is empty where it can continue to schedule immediatly.
Von meinem iPhone gesendet
> Am 05.10.2017 um 23:49 schrieb David P Grove <gr...@us.ibm.com>:
>
> I can see the value in delaying the binding of activations to invokers when the system is loaded (can't execute "immediately" on its target invoker).
>
> Perhaps in ignorance, I am a little worried about the scalability of a single backlog topic. With a few hundred invokers, it seems like we'd be exposed to frequent and expensive partition rebalancing operations as invokers crash/restart. Maybe if we have N = K*M invokers, we can get away with M backlog topics each being read by K invokers. We could still get imbalance across the different backlog topics, but it might be good enough.
>
> I think we'd also need to do some thinking of how to ensure that work put in a backlog topic doesn't languish there for a really long time. Once we start having work in the backlog, do we need to stop putting work in immediately topics? If we do, that could hurt overall performance. If we don't, how will the backlog topic ever get drained if most invokers are kept busy servicing their immediately topics?
>
> --dave
>
> Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invok
>
> From: Tyson Norris <tn...@adobe.com.INVALID>
> To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
> Date: 10/04/2017 07:45 PM
> Subject: Invoker activation queueing proposal
>
>
>
>
> Hi -
>
> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>
>
>
> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>
> - execute the activation "immediately" if capacity is available
>
> - provide a single overflow topic for activations that cannot execute “immediately"
>
> - schedule from the overflow topic when capacity is available
>
>
>
> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>
>
>
> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>
>
>
> I put a wiki page proposal here: https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Fe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk&m=UE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM&s=LD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8&e=
>
>
>
> WDYT?
>
>
>
> Thanks
>
> Tyson
>
>
>
>
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
More on the queueing proposal:
One problem with a single unified overflow topic is the case of multiple controllers WITH shared data, where:
- all controllers are in overflow state
- activation A1 arrives at controller C0, timeout scheduled, message sent to overflow topic
- controller C1 gets a view on invokers with capacity BEFORE controller C0 does - begins processing overflow topic
- activation A1 is now being processed by C1 (but the request and initial timeout is being processed on controller C0)
For this case, I am thinking that it should be possible to use a different controller to process the activation than the controller that originally received the request/activation(!).
To do this:
- the timeout scheduled for for the overflow-processing controller, would consider time already spent waiting for invokers. (A failed completion messages would be sent to the initial controllers completion topic in case of timeout)
- when activation is scheduled to an invoker, the original controller (if different) must be noted in the LoadBalancerData entry
- successful completion messages from invoker would be sent to the processing controller's completion topic (and THEN be sent to the initial controllers completion topic), since both controllers are waiting for timeout of that activation processing in invoker
For the case of multiple controllers WITHOUT shared data: I think the same approach works, so overflow activations would be processed asap by any available controller + any available invoker, as opposed to current controller waiting for its own in-flight activations to complete before overflow is processed. (Only difference is that invokers will be overscheduled due to inability of controllers to view other controllers’ scheduled activations).
I’ll try to come up with a diagram to describe this but wanted to mention it to see if people have feedback on the idea in the meantime.
Thanks
Tyson
> On Oct 10, 2017, at 10:34 AM, Markus Thömmes <ma...@me.com> wrote:
>
> Heyho,
>
> I ran into the same issue before and I think our scheduling code should be an Actor. We could microbenchmark it to assure it can happily schedule a large amount of actions per second to not become a bottleneck.
>
> +1 for actorizing the LB
>
> Cheers,
> Markus
>
> Von meinem iPhone gesendet
>
>> Am 10.10.2017 um 13:28 schrieb Tyson Norris <tn...@adobe.com.INVALID>:
>>
>> Hi -
>> Following up on this, I’ve been working on a PR.
>>
>> One issue I’ve run into (which may be problematic in other scheduling scenarios) is that the scheduling in LoadBalancerService doesn’t respect the new async nature of activation counting in LoadBalancerData. At least I think this is a good description.
>>
>> Specifically, I am creating a test that submits activations via LoadBalancer.publish, and I end up with 10 activations scheduled on invoker0, even though I use an invokerBusyThreshold of 1.
>> It would only occur when concurrent requests (or very short time between?) arrive at the same controller, I think. (Otherwise the counts can sync up quickly enough)
>> I’ll work more on testing it.
>>
>> Assuming this (dealing with async counters) is the problem, I’m not exactly sure how to deal with it. Some options may include:
>> - change LoadBalancer to an actor, so that local counter states can be easier managed (these would still need to replicate, but at least locally it would do the right thing)
>> - coordinate the schedule + setupActivation calls to also rely on some local state for activations that should be counted but have not yet been processed within LoadBalancerData
>>
>> Any suggestions in this area would be great.
>>
>> Thanks
>> Tyson
>>
>>
>>
>>> On Oct 6, 2017, at 11:04 AM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>>
>>> With many invokers, there is less data exposed to rebalancing operations, since the invoker topics will only ever receive enough activations that can be processed “immediately", currently set to 16. The single backlog topic would only be consumed by the controller (not any invoker), and the invokers would only consumer their respective “process immediately” topic - which effectively has no, or very little, backlog - 16 max. My suggestion is that having multiple backlogs is an unnecessary problem, regardless of how many invokers there are.
>>>
>>> It is worth noting the case of multiple controllers as well, where multiple controllers may be processing the same backlog topic. I don’t think this should cause any more trouble than the distributed activation counting that should be enabled via controller clustering, but it may mean that if one controller enters overflow state, it should signal that ALL controllers are now in overflow state, etc.
>>>
>>> Regarding “timeout”, I would plan to use the existing timeout mechanism, where an ActivationEntry is created immediately, regardless of whether the activation is going to get processed, or get added to the backlog. At time of processing the backlog message, if the entry is timed out, throw it away. (The entry map may need to be shared in the case multiple invokers are in use, and they all consume from the same topic; alternatively, we can partition the topic so that entries are only processed by the controller that has backlogged them)
>>>
>>> Yes, once invokers are saturated, and backlogging begins, I think all incoming activations should be sent straight to backlog (we already know that no invokers are available). This should not hurt overall performance anymore than it currently does, and should be better (since the first invoker available can start taking work, instead of waiting on a specific invoker to become available).
>>>
>>> I’m working on a PR, I think much of these details will come out there, but in the meantime, let me know if any of this doesn’t make sense.
>>>
>>> Thanks
>>> Tyson
>>>
>>>
>>> On Oct 5, 2017, at 2:49 PM, David P Grove <gr...@us.ibm.com>> wrote:
>>>
>>>
>>> I can see the value in delaying the binding of activations to invokers when the system is loaded (can't execute "immediately" on its target invoker).
>>>
>>> Perhaps in ignorance, I am a little worried about the scalability of a single backlog topic. With a few hundred invokers, it seems like we'd be exposed to frequent and expensive partition rebalancing operations as invokers crash/restart. Maybe if we have N = K*M invokers, we can get away with M backlog topics each being read by K invokers. We could still get imbalance across the different backlog topics, but it might be good enough.
>>>
>>> I think we'd also need to do some thinking of how to ensure that work put in a backlog topic doesn't languish there for a really long time. Once we start having work in the backlog, do we need to stop putting work in immediately topics? If we do, that could hurt overall performance. If we don't, how will the backlog topic ever get drained if most invokers are kept busy servicing their immediately topics?
>>>
>>> --dave
>>>
>>> Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invok
>>>
>>> From: Tyson Norris <tn...@adobe.com.INVALID>>
>>> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
>>> Date: 10/04/2017 07:45 PM
>>> Subject: Invoker activation queueing proposal
>>>
>>> ________________________________
>>>
>>>
>>>
>>> Hi -
>>>
>>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>>
>>>
>>>
>>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>>
>>> - execute the activation "immediately" if capacity is available
>>>
>>> - provide a single overflow topic for activations that cannot execute “immediately"
>>>
>>> - schedule from the overflow topic when capacity is available
>>>
>>>
>>>
>>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>>
>>>
>>>
>>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>>
>>>
>>>
>>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e&data=02%7C01%7C%7C206a28eb8f9d4f3b131608d50ce4be0f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636429098932393323&sdata=pLsSnlJRYtL4cHMqciGBsA9kLaHzW1GjbijpJCsD1po%3D&reserved=0=<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e%3D&data=02%7C01%7C%7C36a3439a232c45d2119c08d50c3b096b%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636428370067764903&sdata=MBzAhIAVOdHCG0acu8YKCNmeYXO8T9PcILoQrlUyixw%3D&reserved=0>
>>>
>>>
>>>
>>> WDYT?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Tyson
>>>
>>
Re: Invoker activation queueing proposal
Posted by Markus Thömmes <ma...@me.com>.
Heyho,
I ran into the same issue before and I think our scheduling code should be an Actor. We could microbenchmark it to assure it can happily schedule a large amount of actions per second to not become a bottleneck.
+1 for actorizing the LB
Cheers,
Markus
Von meinem iPhone gesendet
> Am 10.10.2017 um 13:28 schrieb Tyson Norris <tn...@adobe.com.INVALID>:
>
> Hi -
> Following up on this, I’ve been working on a PR.
>
> One issue I’ve run into (which may be problematic in other scheduling scenarios) is that the scheduling in LoadBalancerService doesn’t respect the new async nature of activation counting in LoadBalancerData. At least I think this is a good description.
>
> Specifically, I am creating a test that submits activations via LoadBalancer.publish, and I end up with 10 activations scheduled on invoker0, even though I use an invokerBusyThreshold of 1.
> It would only occur when concurrent requests (or very short time between?) arrive at the same controller, I think. (Otherwise the counts can sync up quickly enough)
> I’ll work more on testing it.
>
> Assuming this (dealing with async counters) is the problem, I’m not exactly sure how to deal with it. Some options may include:
> - change LoadBalancer to an actor, so that local counter states can be easier managed (these would still need to replicate, but at least locally it would do the right thing)
> - coordinate the schedule + setupActivation calls to also rely on some local state for activations that should be counted but have not yet been processed within LoadBalancerData
>
> Any suggestions in this area would be great.
>
> Thanks
> Tyson
>
>
>
>> On Oct 6, 2017, at 11:04 AM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>>
>> With many invokers, there is less data exposed to rebalancing operations, since the invoker topics will only ever receive enough activations that can be processed “immediately", currently set to 16. The single backlog topic would only be consumed by the controller (not any invoker), and the invokers would only consumer their respective “process immediately” topic - which effectively has no, or very little, backlog - 16 max. My suggestion is that having multiple backlogs is an unnecessary problem, regardless of how many invokers there are.
>>
>> It is worth noting the case of multiple controllers as well, where multiple controllers may be processing the same backlog topic. I don’t think this should cause any more trouble than the distributed activation counting that should be enabled via controller clustering, but it may mean that if one controller enters overflow state, it should signal that ALL controllers are now in overflow state, etc.
>>
>> Regarding “timeout”, I would plan to use the existing timeout mechanism, where an ActivationEntry is created immediately, regardless of whether the activation is going to get processed, or get added to the backlog. At time of processing the backlog message, if the entry is timed out, throw it away. (The entry map may need to be shared in the case multiple invokers are in use, and they all consume from the same topic; alternatively, we can partition the topic so that entries are only processed by the controller that has backlogged them)
>>
>> Yes, once invokers are saturated, and backlogging begins, I think all incoming activations should be sent straight to backlog (we already know that no invokers are available). This should not hurt overall performance anymore than it currently does, and should be better (since the first invoker available can start taking work, instead of waiting on a specific invoker to become available).
>>
>> I’m working on a PR, I think much of these details will come out there, but in the meantime, let me know if any of this doesn’t make sense.
>>
>> Thanks
>> Tyson
>>
>>
>> On Oct 5, 2017, at 2:49 PM, David P Grove <gr...@us.ibm.com>> wrote:
>>
>>
>> I can see the value in delaying the binding of activations to invokers when the system is loaded (can't execute "immediately" on its target invoker).
>>
>> Perhaps in ignorance, I am a little worried about the scalability of a single backlog topic. With a few hundred invokers, it seems like we'd be exposed to frequent and expensive partition rebalancing operations as invokers crash/restart. Maybe if we have N = K*M invokers, we can get away with M backlog topics each being read by K invokers. We could still get imbalance across the different backlog topics, but it might be good enough.
>>
>> I think we'd also need to do some thinking of how to ensure that work put in a backlog topic doesn't languish there for a really long time. Once we start having work in the backlog, do we need to stop putting work in immediately topics? If we do, that could hurt overall performance. If we don't, how will the backlog topic ever get drained if most invokers are kept busy servicing their immediately topics?
>>
>> --dave
>>
>> Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invok
>>
>> From: Tyson Norris <tn...@adobe.com.INVALID>>
>> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
>> Date: 10/04/2017 07:45 PM
>> Subject: Invoker activation queueing proposal
>>
>> ________________________________
>>
>>
>>
>> Hi -
>>
>> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>>
>>
>>
>> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>>
>> - execute the activation "immediately" if capacity is available
>>
>> - provide a single overflow topic for activations that cannot execute “immediately"
>>
>> - schedule from the overflow topic when capacity is available
>>
>>
>>
>> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>>
>>
>>
>> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>>
>>
>>
>> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e&data=02%7C01%7C%7C206a28eb8f9d4f3b131608d50ce4be0f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636429098932393323&sdata=pLsSnlJRYtL4cHMqciGBsA9kLaHzW1GjbijpJCsD1po%3D&reserved=0=<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e%3D&data=02%7C01%7C%7C36a3439a232c45d2119c08d50c3b096b%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636428370067764903&sdata=MBzAhIAVOdHCG0acu8YKCNmeYXO8T9PcILoQrlUyixw%3D&reserved=0>
>>
>>
>>
>> WDYT?
>>
>>
>>
>> Thanks
>>
>> Tyson
>>
>
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
Hi -
Following up on this, I’ve been working on a PR.
One issue I’ve run into (which may be problematic in other scheduling scenarios) is that the scheduling in LoadBalancerService doesn’t respect the new async nature of activation counting in LoadBalancerData. At least I think this is a good description.
Specifically, I am creating a test that submits activations via LoadBalancer.publish, and I end up with 10 activations scheduled on invoker0, even though I use an invokerBusyThreshold of 1.
It would only occur when concurrent requests (or very short time between?) arrive at the same controller, I think. (Otherwise the counts can sync up quickly enough)
I’ll work more on testing it.
Assuming this (dealing with async counters) is the problem, I’m not exactly sure how to deal with it. Some options may include:
- change LoadBalancer to an actor, so that local counter states can be easier managed (these would still need to replicate, but at least locally it would do the right thing)
- coordinate the schedule + setupActivation calls to also rely on some local state for activations that should be counted but have not yet been processed within LoadBalancerData
Any suggestions in this area would be great.
Thanks
Tyson
> On Oct 6, 2017, at 11:04 AM, Tyson Norris <tn...@adobe.com.INVALID> wrote:
>
> With many invokers, there is less data exposed to rebalancing operations, since the invoker topics will only ever receive enough activations that can be processed “immediately", currently set to 16. The single backlog topic would only be consumed by the controller (not any invoker), and the invokers would only consumer their respective “process immediately” topic - which effectively has no, or very little, backlog - 16 max. My suggestion is that having multiple backlogs is an unnecessary problem, regardless of how many invokers there are.
>
> It is worth noting the case of multiple controllers as well, where multiple controllers may be processing the same backlog topic. I don’t think this should cause any more trouble than the distributed activation counting that should be enabled via controller clustering, but it may mean that if one controller enters overflow state, it should signal that ALL controllers are now in overflow state, etc.
>
> Regarding “timeout”, I would plan to use the existing timeout mechanism, where an ActivationEntry is created immediately, regardless of whether the activation is going to get processed, or get added to the backlog. At time of processing the backlog message, if the entry is timed out, throw it away. (The entry map may need to be shared in the case multiple invokers are in use, and they all consume from the same topic; alternatively, we can partition the topic so that entries are only processed by the controller that has backlogged them)
>
> Yes, once invokers are saturated, and backlogging begins, I think all incoming activations should be sent straight to backlog (we already know that no invokers are available). This should not hurt overall performance anymore than it currently does, and should be better (since the first invoker available can start taking work, instead of waiting on a specific invoker to become available).
>
> I’m working on a PR, I think much of these details will come out there, but in the meantime, let me know if any of this doesn’t make sense.
>
> Thanks
> Tyson
>
>
> On Oct 5, 2017, at 2:49 PM, David P Grove <gr...@us.ibm.com>> wrote:
>
>
> I can see the value in delaying the binding of activations to invokers when the system is loaded (can't execute "immediately" on its target invoker).
>
> Perhaps in ignorance, I am a little worried about the scalability of a single backlog topic. With a few hundred invokers, it seems like we'd be exposed to frequent and expensive partition rebalancing operations as invokers crash/restart. Maybe if we have N = K*M invokers, we can get away with M backlog topics each being read by K invokers. We could still get imbalance across the different backlog topics, but it might be good enough.
>
> I think we'd also need to do some thinking of how to ensure that work put in a backlog topic doesn't languish there for a really long time. Once we start having work in the backlog, do we need to stop putting work in immediately topics? If we do, that could hurt overall performance. If we don't, how will the backlog topic ever get drained if most invokers are kept busy servicing their immediately topics?
>
> --dave
>
> Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invok
>
> From: Tyson Norris <tn...@adobe.com.INVALID>>
> To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
> Date: 10/04/2017 07:45 PM
> Subject: Invoker activation queueing proposal
>
> ________________________________
>
>
>
> Hi -
>
> I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
>
>
>
> In short: Instead of scheduling activations to an invoker on receipt, do the following:
>
> - execute the activation "immediately" if capacity is available
>
> - provide a single overflow topic for activations that cannot execute “immediately"
>
> - schedule from the overflow topic when capacity is available
>
>
>
> (BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
>
>
>
> Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
>
>
>
> I put a wiki page proposal here: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e&data=02%7C01%7C%7C206a28eb8f9d4f3b131608d50ce4be0f%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636429098932393323&sdata=pLsSnlJRYtL4cHMqciGBsA9kLaHzW1GjbijpJCsD1po%3D&reserved=0=<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e%3D&data=02%7C01%7C%7C36a3439a232c45d2119c08d50c3b096b%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636428370067764903&sdata=MBzAhIAVOdHCG0acu8YKCNmeYXO8T9PcILoQrlUyixw%3D&reserved=0>
>
>
>
> WDYT?
>
>
>
> Thanks
>
> Tyson
>
Re: Invoker activation queueing proposal
Posted by Tyson Norris <tn...@adobe.com.INVALID>.
With many invokers, there is less data exposed to rebalancing operations, since the invoker topics will only ever receive enough activations that can be processed “immediately", currently set to 16. The single backlog topic would only be consumed by the controller (not any invoker), and the invokers would only consumer their respective “process immediately” topic - which effectively has no, or very little, backlog - 16 max. My suggestion is that having multiple backlogs is an unnecessary problem, regardless of how many invokers there are.
It is worth noting the case of multiple controllers as well, where multiple controllers may be processing the same backlog topic. I don’t think this should cause any more trouble than the distributed activation counting that should be enabled via controller clustering, but it may mean that if one controller enters overflow state, it should signal that ALL controllers are now in overflow state, etc.
Regarding “timeout”, I would plan to use the existing timeout mechanism, where an ActivationEntry is created immediately, regardless of whether the activation is going to get processed, or get added to the backlog. At time of processing the backlog message, if the entry is timed out, throw it away. (The entry map may need to be shared in the case multiple invokers are in use, and they all consume from the same topic; alternatively, we can partition the topic so that entries are only processed by the controller that has backlogged them)
Yes, once invokers are saturated, and backlogging begins, I think all incoming activations should be sent straight to backlog (we already know that no invokers are available). This should not hurt overall performance anymore than it currently does, and should be better (since the first invoker available can start taking work, instead of waiting on a specific invoker to become available).
I’m working on a PR, I think much of these details will come out there, but in the meantime, let me know if any of this doesn’t make sense.
Thanks
Tyson
On Oct 5, 2017, at 2:49 PM, David P Grove <gr...@us.ibm.com>> wrote:
I can see the value in delaying the binding of activations to invokers when the system is loaded (can't execute "immediately" on its target invoker).
Perhaps in ignorance, I am a little worried about the scalability of a single backlog topic. With a few hundred invokers, it seems like we'd be exposed to frequent and expensive partition rebalancing operations as invokers crash/restart. Maybe if we have N = K*M invokers, we can get away with M backlog topics each being read by K invokers. We could still get imbalance across the different backlog topics, but it might be good enough.
I think we'd also need to do some thinking of how to ensure that work put in a backlog topic doesn't languish there for a really long time. Once we start having work in the backlog, do we need to stop putting work in immediately topics? If we do, that could hurt overall performance. If we don't, how will the backlog topic ever get drained if most invokers are kept busy servicing their immediately topics?
--dave
Tyson Norris ---10/04/2017 07:45:38 PM---Hi - I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invok
From: Tyson Norris <tn...@adobe.com.INVALID>>
To: "dev@openwhisk.apache.org<ma...@openwhisk.apache.org>" <de...@openwhisk.apache.org>>
Date: 10/04/2017 07:45 PM
Subject: Invoker activation queueing proposal
________________________________
Hi -
I’ve been discussing a bit with a few about optimizing the queueing that goes on ahead of invokers so that things behave more simply and predictable.
In short: Instead of scheduling activations to an invoker on receipt, do the following:
- execute the activation "immediately" if capacity is available
- provide a single overflow topic for activations that cannot execute “immediately"
- schedule from the overflow topic when capacity is available
(BTW “Immediately” means: still queued via existing invoker topics, but ONLY gets queued there in the case that the invoker is not fully loaded, and therefore should execute it “very soon")
Later: it would also be good to provide more container state data from invoker to controller, to get better scheduling options - e.g. if some invokers can handle running more containers than other invokers, that info can be used to avoid over/under-loading the invokers (currently we assume each invoker can handle 16 activations, I think)
I put a wiki page proposal here: https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Fe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk&m=UE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM&s=LD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8&e=<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange%26d%3DDwIGaQ%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3DFe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk%26m%3DUE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM%26s%3DLD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8%26e%3D&data=02%7C01%7C%7C36a3439a232c45d2119c08d50c3b096b%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636428370067764903&sdata=MBzAhIAVOdHCG0acu8YKCNmeYXO8T9PcILoQrlUyixw%3D&reserved=0>
WDYT?
Thanks
Tyson
Re: Invoker activation queueing proposal
Posted by David P Grove <gr...@us.ibm.com>.
I can see the value in delaying the binding of activations to invokers when
the system is loaded (can't execute "immediately" on its target invoker).
Perhaps in ignorance, I am a little worried about the scalability of a
single backlog topic. With a few hundred invokers, it seems like we'd be
exposed to frequent and expensive partition rebalancing operations as
invokers crash/restart. Maybe if we have N = K*M invokers, we can get away
with M backlog topics each being read by K invokers. We could still get
imbalance across the different backlog topics, but it might be good enough.
I think we'd also need to do some thinking of how to ensure that work put
in a backlog topic doesn't languish there for a really long time. Once we
start having work in the backlog, do we need to stop putting work in
immediately topics? If we do, that could hurt overall performance. If we
don't, how will the backlog topic ever get drained if most invokers are
kept busy servicing their immediately topics?
--dave
From: Tyson Norris <tn...@adobe.com.INVALID>
To: "dev@openwhisk.apache.org" <de...@openwhisk.apache.org>
Date: 10/04/2017 07:45 PM
Subject: Invoker activation queueing proposal
Hi -
I’ve been discussing a bit with a few about optimizing the queueing that
goes on ahead of invokers so that things behave more simply and
predictable.
In short: Instead of scheduling activations to an invoker on receipt, do
the following:
- execute the activation "immediately" if capacity is available
- provide a single overflow topic for activations that cannot execute
“immediately"
- schedule from the overflow topic when capacity is available
(BTW “Immediately” means: still queued via existing invoker topics, but
ONLY gets queued there in the case that the invoker is not fully loaded,
and therefore should execute it “very soon")
Later: it would also be good to provide more container state data from
invoker to controller, to get better scheduling options - e.g. if some
invokers can handle running more containers than other invokers, that info
can be used to avoid over/under-loading the invokers (currently we assume
each invoker can handle 16 activations, I think)
I put a wiki page proposal here:
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_OPENWHISK_Invoker-2BActivation-2BQueueing-2BChange&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Fe4FicGBU_20P2yihxV-apaNSFb6BSj6AlkptSF2gMk&m=UE8OIR_GnMltmRZyIuLVHMlzyQvNku-H7kLk67u45IM&s=LD75-npfzA7qzUGNgYbFBy4qKatnkdO5I2vKYSGUBg8&e=
WDYT?
Thanks
Tyson