You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2021/09/02 18:25:06 UTC
[GitHub] [openwhisk] aFuerst opened a new issue #5154: Invoker marked Unhealthy but no health check failure in logs
aFuerst opened a new issue #5154:
URL: https://github.com/apache/openwhisk/issues/5154
## Environment details:
Running using ansible local setup, with 1 invoker
Docker version 20.10.8, build 3967b7d28e
Ubuntu 20.10
## Steps to reproduce the issue:
Run using Ansible
Create a simple python action
Try to invoke that action
## Provide the expected results and outputs:
Action runs
## Provide the actual results and outputs:
```
error: Unable to invoke action 'test': The server is currently unavailable (because it is overloaded or down for maintenance). (code c62639068253a7d3d1ff9fa45d9327c1)
```
## Additional information you deem important:
The invoker is running correctly, gets the health test action, and returns an ACK.
But for some reason is declared unhealthy.
No errors or action failures.
Relevant Invoker logs
```
[2021-09-02T17:58:26.795Z] [WARN] [#tid_sid_invokerHealth] [InvokerReactive] revision was not provided for whisk.system/invokerHealthTestAction0
[INFO] [#tid_sid_invokerHealth] [CouchDbRestStore] [GET] 'whisk_local_whisks' finding document: 'id: whisk.system/invokerHealthTestAction0' [marker:database_getDocument_start:5
45597]
[2021-09-02T17:58:26.917Z] [INFO] [#tid_sid_invokerHealth] [CouchDbRestStore] [marker:database_getDocument_finish:545718:121]
[2021-09-02T17:58:26.920Z] [INFO] [#tid_sid_invokerHealth] [CouchDbRestStore] [PUT] 'whisk_local_activations' saving document: 'id: whisk.system/2179ea997f664c69b9ea997f66ec6900, rev: null' [marker:datab
ase_saveDocument_start:545722]
[2021-09-02T17:58:26.921Z] [INFO] [#tid_sid_dbBatcher] [CouchDbRestStore] 'whisk_local_activations' saving 1 documents [marker:database_saveDocumentBulk_start:416615]
[2021-09-02T17:58:26.925Z] [INFO] [#tid_sid_invokerHealth] [MessagingActiveAck] posted combined of activation 2179ea997f664c69b9ea997f66ec690
```
Relevant Controller logs
```
2021-09-02T17:49:26.740Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] registered a new invoker: invoker0
[2021-09-02T17:49:26.756Z] [INFO] [#tid_sid_invokerHealth] [InvokerActor] invoker0 is unhealthy [marker:loadbalancer_invokerState.unhealthy_counter:5557]
[2021-09-02T17:49:26.769Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '4ac9a7b744d64b4089a7b744d66b406f' [marker:controller_kafka_start:55
71]
[2021-09-02T17:49:26.781Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] loadbalancer invoker status updated. managedInvokers = 1 blackboxInvokers = 1. number of known invokers incr
eased: new = 1, old = 0. details: invoker0/0: Unhealthy / 2048 MB of 2048 MB.
[2021-09-02T17:49:26.782Z] [INFO] [#tid_sid_invokerHealth] [InvokerPool] invoker status changed to 0 -> Unhealthy
[2021-09-02T17:58:26.783Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posting topic 'invoker0' with activation id '2179ea997f664c69b9ea997f66ec6900' [marker:controller_kafka_start:545585]
[2021-09-02T17:58:26.789Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] posted to invoker0[0][7] [marker:controller_kafka_finish:545591:6]
[2021-09-02T17:58:26.928Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received completion ack for health action on invoker0/0
[2021-09-02T17:58:26.928Z] [INFO] [#tid_sid_invokerHealth] [ShardingContainerPoolBalancer] received result ack for '2179ea997f664c69b9ea997f66ec6900'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] aFuerst commented on issue #5154: Invoker marked Unhealthy but no health check failure in logs
Posted by GitBox <gi...@apache.org>.
aFuerst commented on issue #5154:
URL: https://github.com/apache/openwhisk/issues/5154#issuecomment-912575341
I was able to figure out what was causing the health check to fail. My override to the memory configuration set the `min` value too small for the nodejs container to work.
I had to change [this line](https://github.com/apache/openwhisk/blob/master/core/controller/src/main/scala/org/apache/openwhisk/core/loadBalancer/InvokerSupervision.scala#L253) to use a different value.
The default min is 128 already, so I did this: `limits = ActionLimits(memory = MemoryLimit(ByteSize(128, SizeUnits.MB))))`
Config I used in question:
```
whisk{
memory {
min = 60 m
max = 1024 m
std = 512 m
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] rabbah commented on issue #5154: Invoker marked Unhealthy but no health check failure in logs
Posted by GitBox <gi...@apache.org>.
rabbah commented on issue #5154:
URL: https://github.com/apache/openwhisk/issues/5154#issuecomment-912680869
👍 @aFuerst, glad you got it resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [openwhisk] aFuerst closed issue #5154: Invoker marked Unhealthy but no health check failure in logs
Posted by GitBox <gi...@apache.org>.
aFuerst closed issue #5154:
URL: https://github.com/apache/openwhisk/issues/5154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@openwhisk.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org