[GitHub] [openwhisk] chetanmeh opened a new pull request #4632: Activation Persister Service

[GitBox <gi...@apache.org>]

chetanmeh opened a new pull request #4632: Activation Persister Service
URL: https://github.com/apache/openwhisk/pull/4632
 
 
   This PR introduces a new micro service - `Activation Persister Service`. This is based on discussion done [here][1] and [here][2]. 
   
   ## Description
   
   This service fullfills following objective
   
   1. Move out the load of persisting Activation results from Invoker to a new service
   2. Enables a more controlled write to backend db in case rate of production of activation is more than the rate at which db can persist them within given capacity
   
   With this service the activation results would now instead written to a Kafka topic and then picked by this service and then saved in db. The approach taken here varies with following parameters
   
   1. Is the activation for a blocking or non blocking call
   2. Is the activation result being persisted from Controller (for sequence/composition) or from Invoker
   3. Does the setup stores logs as part of activation or store them seprately
   
   The implementation consist of following high level components
   
   1. Activation Persister Service - A new service impl in `core/persister` module. It is based on [Alpakka Kafka][3] and reads the activation record from Kafka and saves them to db
   2. `KafkaActivationStore` - An extension of `ArtifactActivationStore` which also routes to Kafka via `MessageProducer` connector
   3. InvokerReactive - The `send` logic would be modfied (details below)
   4. New Kafka topic `completed-non-blocking`
   
   Some key aspects which impact the implementation are
   
   1. Blocking/Non Blocking - Depending on invocation type the nature of activation result sent via Kafka `completed` topic varies
       a. Blocking - The `ResultMessage` contains the activation result without logs
       b. Non Blocking - Only `CompletionMessage` is sent
   
   ### Activation Persister Service
   
   This service implements a streaming flow based on [Alpakka Kafka][3]. It runs in 2 modes
   
   #### ActiationResult without Logs* 
   
   If the setup does not store the activation with logs then persister service would listen for `ResultMessage` on all `completed.*` topics. This would also pickup records from `completed-non-blocking` topic.
   
   Further `KafkaActivationStore` behaviour would change depending on where its being used
   
   1. Within `Controller` - Store all activations to `completed-non-blocking` (??)
   2. Within `Invoker` - Noop. It would just drop all `store` calls 
   
   InvokerReactive send flow would send all WhiskActivation result for blocking calls to existing `completed<instanceId>` topic. In addition non blocking results would also get routed to Kafka to `completed-non-blocking`
   
   #### ActiationResult with Logs
   
   TBD
   
   [1]: https://lists.apache.org/thread.html/7c14d2075e4f8258ec6e909f6b443f4b899b9982103a29b520d963ac@%3Cdev.openwhisk.apache.org%3E
   [2]:https://lists.apache.org/thread.html/0f51397cfe6a910e67b8400f8dbbf07ea46dddd8b8c3b31c771510a8@%3Cdev.openwhisk.apache.org%3E
   [3]: https://doc.akka.io/docs/alpakka-kafka/current/home.html
   
   ## Related issue and scope
   <!--- Please include a link to a related issue if there is one. -->
   - [ ] I opened an issue to propose and discuss this change (#????)
   
   ## My changes affect the following components
   <!--- Select below all system components are affected by your change. -->
   <!--- Enter an `x` in all applicable boxes. -->
   - [ ] API
   - [ ] Controller
   - [ ] Message Bus (e.g., Kafka)
   - [ ] Loadbalancer
   - [ ] Invoker
   - [ ] Intrinsic actions (e.g., sequences, conductors)
   - [ ] Data stores (e.g., CouchDB)
   - [ ] Tests
   - [ ] Deployment
   - [ ] CLI
   - [ ] General tooling
   - [ ] Documentation
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
   - [ ] Bug fix (generally a non-breaking change which closes an issue).
   - [ ] Enhancement or new feature (adds new functionality).
   - [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
   
   ## Checklist:
   <!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
   
   - [ ] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
   - [ ] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
   - [ ] I added tests to cover my changes.
   - [ ] My changes require further changes to the documentation.
   - [ ] I updated the documentation where necessary.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services