You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2022/10/23 07:46:03 UTC
[GitHub] [openwhisk] style95 commented on pull request #5338: [WIP] Add zero downtime deployment
style95 commented on PR #5338:
URL: https://github.com/apache/openwhisk/pull/5338#issuecomment-1288042098
I could successfully test this feature.

When a new version of a scheduler and an invoker are newly deployed, there is no queue and container.
So when a new version of a controller is deployed, it should create a queue and a container first accordingly wait time increases.
But every single activation was successfully invoked without any failure during the deployment.
We can see no error happened in the above screenshot.
Regarding the wait-time increase, I think it's orthogonal to this PR and we can improve it in a subsequent PR.
**Controllers**
```
[2022-10-23T06:05:01.123Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] POST /api/v1/namespaces/guest/actions/noop30 blocking=true&result=true&volatile=true
[2022-10-23T06:05:01.182Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] posted to scheduler0[0][51090548]
[2022-10-23T06:05:01.182Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] Created queue successfully for guest/guest/noop30@0.0.1 on scheduler0
[2022-10-23T06:05:08.430Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] received completion ack for '69a85a8cd2294ea5a85a8cd229aea531', system error=false
[2022-10-23T06:05:08.433Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FPCPoolBalancer] received result ack for '69a85a8cd2294ea5a85a8cd229aea531'
```
**schedulers**
```
[2022-10-23T06:05:01.189Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [QueueManager] Got activation message 69a85a8cd2294ea5a85a8cd229aea531 for Namespace(guest,23bc46b1-71f6-4ed5-8c54-816aa4f8c502)/guest/noop30@0.0.1 from kafka.
[2022-10-23T06:05:01.190Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MemoryQueue] [guest:guest/noop30@0.0.1:Running] got a new activation message 69a85a8cd2294ea5a85a8cd229aea531
[2022-10-23T06:05:01.302Z] [INFO] [#tid_sid_unknown] [SchedulingDecisionMaker] add one container if totalContainers(0) == 0 && availableMsg(1) > 0 [guest:guest/noop30@0.0.1]
[2022-10-23T06:05:01.302Z] [INFO] [#tid_sid_unknown] [MemoryQueue] [guest:guest/noop30@0.0.1:Running] Try to create a new container with creationId e3182d5e23a34ed9982d5e23a3aed9f3
[2022-10-23T06:05:01.303Z] [INFO] [#tid_sid_unknown] [ContainerManager] received 1 creation message [guest:guest/noop30@0.0.1]
[2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] usable total invoker size: 4
[2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] 1 creation messages for guest/guest/noop30@0.0.1, managedFraction:1.0, blackboxFraction:1.0, managed invoker size:4, blackboxes invoker size:4
[2022-10-23T06:05:01.306Z] [INFO] [#tid_sid_unknown] [ContainerManager] [guest/guest/noop30@0.0.1] looking for managed invokers to schedule.
[2022-10-23T06:05:01.309Z] [INFO] [#tid_sid_containerCreation] [ContainerManager] posted creationId: e3182d5e23a34ed9982d5e23a3aed9f3 for guest/guest/noop30@0.0.1 to invoker0[0][13720]
[2022-10-23T06:05:04.233Z] [INFO] [#tid_IH8hZVMxCVzB5xW7KcMsgGiSr6X81Hz9] [MemoryQueue] [guest:guest/noop30@0.0.1:Running] Get activation request 431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f, send one message: 69a85a8cd2294ea5a85a8cd229aea531
```
**Invokers**
```
[2022-10-23T06:05:02.780Z] [INFO] [#tid_sid_unknown] [FunctionPullingContainerPool] received a container creation message: e3182d5e23a34ed9982d5e23a3aed9f3
[2022-10-23T06:05:02.828Z] [INFO] [#tid_sid_invokerColdstart] [DockerClientWithFileAccess] running /usr/bin/docker run -d --cpu-shares 25 --memory 256m --memory-swap 256m --network bridge -e __OW_API_HOST=https://10.106.219.110 -e __OW_ALLOW_CONCURRENT=false --name wsk00_19_guest_noop30 --cap-drop NET_RAW --cap-drop NET_ADMIN --ulimit nofile=1024:1024 --pids-limit 1024 --log-driver json-file openwhisk/action-nodejs-v14:nightly (timeout: 1 minute)
[2022-10-23T06:05:04.224Z] [INFO] [#tid_sid_unknown] [WatcherService] watch endpoint: WatchEndpoint(bmt-new-110/namespace/guest/guest/noop30/15-e0e9a79b4d38b0b010f35023a7464ef6/invoker0/container/431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f,,false,data-management-service,Set(DeleteEvent))
[2022-10-23T06:05:04.227Z] [INFO] [#tid_sid_unknown] [FPCInvokerReactive] Posted success ack of container creation e3182d5e23a34ed9982d5e23a3aed9f3 for guest/guest/noop30@0.0.1
[2022-10-23T06:05:04.240Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [FunctionPullingContainerProxy] received a message 69a85a8cd2294ea5a85a8cd229aea531 for guest/noop30@0.0.1 in ClientCreated
[2022-10-23T06:05:04.240Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] sending initialization to ContainerId(431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f) ContainerAddress(172.17.0.21,8080)
[2022-10-23T06:05:05.087Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] initialization result: ok
[2022-10-23T06:05:05.087Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] sending arguments to /guest/noop30 at ContainerId(431fdb1966598b9299d4905ac6cc76a223032352eabe06b667e9166c6671126f) ContainerAddress(172.17.0.21,8080)
[2022-10-23T06:05:05.096Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [DockerContainer] running result: ok
[2022-10-23T06:05:05.099Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MessagingActiveAck] posted result of activation 69a85a8cd2294ea5a85a8cd229aea531
[2022-10-23T06:05:05.100Z] [INFO] [#tid_efb449d34d9fe0f7ce3afea3222e2338] [MessagingActiveAck] posted completion of activation 69a85a8cd2294ea5a85a8cd229aea531
```
As per the above logs, we can see that it took around 200ms for the controller to send a queue creation and the scheduler to send a container creation request. But it took around 1.4s for invokers to receive the creation request, it took another 1.4s for the invoker to create a container. Another thing is it took around 3.3s for the controller to actually receive the activation ack after the invoker sent the acks.
I think this is still an improvement point especially regarding Kakfa messing.
Anyway, I have achieved what I wanted, I would include https://github.com/apache/openwhisk/pull/5334 into this PR and update all CI tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org