[GitHub] style95 commented on issue #2360: Choose target invoker based on specific invoker load.

[gi...@git.apache.org]

style95 commented on issue #2360: Choose target invoker based on specific invoker load.
URL: https://github.com/apache/incubator-openwhisk/pull/2360#issuecomment-307989249
 
 
   I hope this could be addressed too.
   Situation is as follow:
   
   1. I have two running invokers.
   2. User sends action invocation request.
   3. At the same time, one invoker which is in charge of invoking the request is down.
   4. It takes some time(5~10 secs) for controller to catch up the situation(invoker is down).
   5. Accordingly, controller tries to send request to the abnormal invoker.
   6. Controller can not receive proper response.
   7. (At this point, I expect request will be automatically redirected to normal invoker but) Controller respond with failed response to client though there is running invoker.
   
   This is because currently, there are kafka topics for each invokers.
   And controller regularly checks the health of invokers with health topic.
   So until the next check, controller will recognize the unhealthy invoker as health invoker.
   
   Because of this, even though we deploy multiple invokers, we can not make invokers high available.
   
   
   I think there are two options.
   
   1. Make invokers to share one kafka topic.
   2. Make controller(or loadbalancer) redirect the request to healthy invoker in case of failure.
   
   
   **1. Make invokers to share one kafka topic.**
   If we can make invokers to share one kafka topic, this situation could be handled.
   Then each invokers will be grouped as one consumer group.
   In case of failure of a target invoker, request will be automatically redirected to healthy invoker at kafka layer.
   
   AFAIK, same request should be sent to same invoker to take advantage of warm container, so there are topics for each invoker.
   So if we take this approach, any invokers should be able to handle any warm containers(which is not created by themselves).
   Even though different invoker receives the same request, the invoker could activate warm container again.
   
   I am not sure this could be easily achievable with current implementation.
   
   
   **2. Controller(or loadbalancer) redirect the request to healthy invoker in case of failure.**
   This approach is to make controller do what kafka does.
   In case of failure, controller should redirect the request to healthy invoker rather than just respond with fail response.
   
   Since many things are already considered in Kafka side, I am inclined to first option.
   
   
   I want you guys to check on this issue too.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services