You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/10/25 01:40:59 UTC
[GitHub] wcorbett36 opened a new issue #324: getsockopt: connection refused
causing liveness probe fail causing controller crashloop
wcorbett36 opened a new issue #324: getsockopt: connection refused causing liveness probe fail causing controller crashloop
URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/324
I am working through a helm deploy on minikube windows 10. I am getting stuck with a crashloop error on my controller. Pods and controller describe posted below. Please advise. Thanks,
λ kubectl get pods -n openwhisk
NAME READY STATUS RESTARTS AGE
apigateway-7b87dd957f-t8j2f 1/1 Running 0 37m
controller-0 0/1 CrashLoopBackOff 15 37m
couchdb-6b87ccfb78-frvbj 1/1 Running 0 37m
init-couchdb-p4wt7 0/1 Completed 0 37m
install-routemgmt-n8xp9 0/1 Init:0/1 0 37m
invoker-6wtwl 0/1 Init:1/2 0 37m
kafka-0 1/1 Running 0 37m
nginx-598c75d7d6-wt25s 1/1 Running 0 37m
redis-5d77674f65-7rq75 1/1 Running 0 37m
zookeeper-0 1/1 Running 0 37m
λ kubectl describe pods/controller-0 -n openwhisk
Name: controller-0
Namespace: openwhisk
Node: minikube/192.168.86.187
Start Time: Wed, 24 Oct 2018 21:00:46 -0400
Labels: controller-revision-hash=controller-69db44c56b
name=controller
statefulset.kubernetes.io/pod-name=controller-0
Annotations: <none>
Status: Running
IP: 172.17.0.8
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID: docker://03f69f049f2cf9a112109933560e69baab1720f354271b742fa5c893e1a05257
Image: busybox
Image ID: docker-pullable://busybox@sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 24 Oct 2018 21:00:52 -0400
Finished: Wed, 24 Oct 2018 21:01:36 -0400
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v7kt2 (ro)
wait-for-couchdb:
Container ID: docker://e6de73d420993fde9cdac225ba622bae22af8400fab50cf1fa69376bdb7188bd
Image: busybox
Image ID: docker-pullable://busybox@sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 24 Oct 2018 21:01:39 -0400
Finished: Wed, 24 Oct 2018 21:05:32 -0400
Ready: True
Restart Count: 0
Environment:
READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v7kt2 (ro)
Containers:
controller:
Container ID: docker://363345272c48f82a50a4261a1433dd3e965bcba389d7b91e3300eb25b4797a92
Image: openwhisk/controller:latest
Image ID: docker-pullable://openwhisk/controller@sha256:79b537a8f9aad7f799d7ff81e9e5a1bd6c74d329ae324ecf792c2a2b7c441dce
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | cut -d'-' -f2`
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Wed, 24 Oct 2018 21:24:02 -0400
Finished: Wed, 24 Oct 2018 21:24:25 -0400
Ready: False
Restart Count: 11
Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
PORT: 8080
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092
KAFKA_HOST_PORT: 9092
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v7kt2 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
ow-core-token-v7kt2:
Type: Secret (a volume populated by a Secret)
SecretName: ow-core-token-v7kt2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 24m default-scheduler Successfully assigned controller-0 to minikube
Normal SuccessfulMountVolume 24m kubelet, minikube MountVolume.SetUp succeeded for volume "ow-core-token-v7kt2"
Normal Pulled 24m kubelet, minikube Container image "busybox" already present on machine
Normal Created 23m kubelet, minikube Created container
Normal Started 23m kubelet, minikube Started container
Normal Pulled 23m kubelet, minikube Container image "busybox" already present on machine
Normal Created 23m kubelet, minikube Created container
Normal Started 23m kubelet, minikube Started container
Warning Unhealthy 18m (x5 over 19m) kubelet, minikube Liveness probe failed: Get http://172.17.0.8:8080/ping: dial tcp 172.17.0.8:8080: getsockopt: connection refused
Normal Pulling 18m (x3 over 19m) kubelet, minikube pulling image "openwhisk/controller:latest"
Normal Pulled 18m (x3 over 19m) kubelet, minikube Successfully pulled image "openwhisk/controller:latest"
Normal Killing 18m (x2 over 18m) kubelet, minikube Killing container with id docker://controller:Container failed liveness probe.. Container will be killed and recreated.
Normal Created 18m (x3 over 19m) kubelet, minikube Created container
Normal Started 18m (x3 over 19m) kubelet, minikube Started container
Warning BackOff 3m (x56 over 17m) kubelet, minikube Back-off restarting failed containe
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services