[GitHub] ddragosd opened a new issue #2263: Controller should not assume DB is initialized when it starts

[gi...@git.apache.org]

ddragosd opened a new issue #2263: Controller should not assume DB is initialized when it starts
URL: https://github.com/apache/incubator-openwhisk/issues/2263
 
 
   This issue came up recently and I'm opening up the discussion here so that we take a decision if we want to fix this or not. I think that in a distributed environment, with a system that depends on other components, making assumptions about the state of other components is not going to work all the time. 
   
   Such is the case with the Controller and the DB. In a recent PR we introduced a nice way for the controller to health-check invokers by asking them to execute a JS action. This only works if the DB is initialized before the controller starts. If that's not the case then the controller is like in a dead-lock mode; it tries to ping invokers, invokers complain they can't fetch the action, and in this time the controller doesn't verify that the action to be executed exists in the DB.
   
   As a result we could have healthy invokers, healthy controllers, but all invocations fail b/c the controller believes there are no healthy invokers available. 
   
   A good way to test this is to spin up [docker-compose](https://github.com/apache/incubator-openwhisk-devtools) . `docker-compose` starts the Controller and the DB about the same time, but the DB takes longer to be initialized, and by the time the ansible script finished, it is already too late for the controller. But we could think of other cases when the DB has the action initially only, but after a while, due to a change, the action `invokerHealthTestAction` is mistakenly deleted, and we're back in the same situation. 
   
   WDYT ?
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services