[GitHub] [openwhisk] tysonnorris opened a new pull request #4698: Add healthcheck for Invoker -> Action Container

[GitBox <gi...@apache.org>]

tysonnorris opened a new pull request #4698: Add healthcheck for Invoker -> Action Container 
URL: https://github.com/apache/openwhisk/pull/4698
 
 
   To validate state of containers before use.
   
   
   <!--- Provide a concise summary of your changes in the Title -->
   
   ## Description
   In our deployment case (mesos/kubernetes) there are occasional cases where prewarm/warm containers fail outside of the usage of ContainerProxy, but ContainerProxy/ContainerPool are not aware of these failures. When this happens, when eventually the containers are used, there is a guarantee of failing the activation, since ContainerProxy assumes the prewarm/warm container is running fine, and it just sees a failure for /init or /run. 
   
   This may be more pertinent to ContainerFactory impls that launch containers on hosts other than the invoker (mesos, k8s, yarn), but since it is also possible for docker containers local to invoker to be interrupted by some other system, it may be useful in all cases as a preventative measure. 
   
   This PR introduces a tcp connection health check from invoker to each action container where:
   - prewarm containers will be connected periodically
   - failure to connect to prewarm will eventually (after configurable number of failures) cause the container to be destroyed and deplaced with a new prewarm container
   - warm containers (paused) will be connected immediately after resume, and immediately before usage for /run
   - failure to connect to warm will cause the job to be rescheduled and the warm container to be destroyed
   
   <!--- Provide a detailed description of your changes. -->
   <!--- Include details of what problem you are solving and how your changes are tested. -->
   
   ## Related issue and scope
   <!--- Please include a link to a related issue if there is one. -->
   - [ ] I opened an issue to propose and discuss this change (#????)
   
   ## My changes affect the following components
   <!--- Select below all system components are affected by your change. -->
   <!--- Enter an `x` in all applicable boxes. -->
   - [ ] API
   - [ ] Controller
   - [ ] Message Bus (e.g., Kafka)
   - [ ] Loadbalancer
   - [x] Invoker
   - [ ] Intrinsic actions (e.g., sequences, conductors)
   - [ ] Data stores (e.g., CouchDB)
   - [ ] Tests
   - [ ] Deployment
   - [ ] CLI
   - [ ] General tooling
   - [ ] Documentation
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
   - [ ] Bug fix (generally a non-breaking change which closes an issue).
   - [x] Enhancement or new feature (adds new functionality).
   - [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
   
   ## Checklist:
   <!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
   
   - [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
   - [x] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
   - [x] I added tests to cover my changes.
   - [ ] My changes require further changes to the documentation.
   - [ ] I updated the documentation where necessary.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services