You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/08/10 15:47:52 UTC
[GitHub] ygalashov commented on issue #272: Controller crash loop
ygalashov commented on issue #272: Controller crash loop
URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/272#issuecomment-412124179
Yes it's failing liveness probe, here is describe output:
```
Name: controller-0
Namespace: openwhisk
Node: gke-cluster-eu-default-pool-3ca68bf7-ph58/10.156.0.2
Start Time: Fri, 10 Aug 2018 18:33:24 +0300
Labels: controller-revision-hash=controller-757c6bf95b
name=controller
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"openwhisk","name":"controller","uid":"b8112e6f-9cb2-11e8-89e0-42010a9c0121","api...
Status: Running
IP: 10.12.0.236
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID: docker://4dbe9a5a8af0d767c8c3679b6bb0194fb1383c800a253622a868d96b62c33203
Image: busybox
Image ID: docker-pullable://busybox@sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; fi; echo waiting for zookeeper to be ready; sleep 1; done; echo zookeeper is up, sleeping for 10 seconds; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 10 Aug 2018 18:33:37 +0300
Finished: Fri, 10 Aug 2018 18:34:08 +0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro)
wait-for-couchdb:
Container ID: docker://f53bf75775348020d4269c4b5c0216148938adb78be9340c43a5edcd92f76a5e
Image: busybox
Image ID: docker-pullable://busybox@sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo verifying CouchDB readiness; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; sleep 1; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 10 Aug 2018 18:34:10 +0300
Finished: Fri, 10 Aug 2018 18:35:52 +0300
Ready: True
Restart Count: 0
Environment:
READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/test_activations
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro)
Containers:
controller:
Container ID: docker://52fa02af0e05fa671606a6cc29fa003609dfe6298d2a0d04ebb49fe8e245e50d
Image: openwhisk/controller:latest
Image ID: docker-pullable://openwhisk/controller@sha256:797fd17bea1894063293640e88de5deb439a7fd0e714de15596b0896aebf5345
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Command:
/bin/bash
-c
/init.sh `hostname | cut -d'-' -f2`
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Fri, 10 Aug 2018 18:40:36 +0300
Finished: Fri, 10 Aug 2018 18:40:54 +0300
Ready: False
Restart Count: 6
Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
PORT: 8080
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
CONFIG_whisk_loadbalancer_invokerBusyThreshold: 8
KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092
KAFKA_HOST_PORT: 9092
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
ow-core-token-zhcqk:
Type: Secret (a volume populated by a Secret)
SecretName: ow-core-token-zhcqk
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned controller-0 to gke-cluster-eu-default-pool-3ca68bf7-ph58
Normal SuccessfulMountVolume 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 MountVolume.SetUp succeeded for volume "ow-core-token-zhcqk"
Normal Pulled 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Container image "busybox" already present on machine
Normal Created 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container
Normal Started 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container
Normal Pulled 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Container image "busybox" already present on machine
Normal Created 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container
Normal Started 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container
Normal Started 10m (x2 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container
Normal Pulling 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 pulling image "openwhisk/controller:latest"
Warning Unhealthy 9m (x5 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Liveness probe failed: Get http://10.12.0.236:8080/ping: dial tcp 10.12.0.236:8080: getsockopt: connection refused
Normal Killing 9m (x2 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Killing container with id docker://controller:Container failed liveness probe.. Container will be killed and recreated.
Normal Pulled 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Successfully pulled image "openwhisk/controller:latest"
Normal Created 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container
Warning FailedSync 8m (x4 over 8m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Error syncing pod
```
As far as I understand liveness probe calls pod directly, not through external ip?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services