You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/11/02 17:12:12 UTC
[GitHub] wcorbett36 commented on issue #324: getsockopt: connection refused
causing liveness probe fail causing controller crashloop
wcorbett36 commented on issue #324: getsockopt: connection refused causing liveness probe fail causing controller crashloop
URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/324#issuecomment-435448024
Hi thanks but the command did not seem to work. My issue seems very similar to:
https://github.com/apache/incubator-openwhisk-deploy-kube/issues/324
invoker-x7tjc 1/1 Running 0 19m
apigateway-7b87dd957f-6qlg9 1/1 Running 0 19m
couchdb-6b87ccfb78-jt54g 1/1 Running 0 19m
nginx-598c75d7d6-sl49v 1/1 Running 0 19m
redis-5d77674f65-2nhzc 1/1 Running 0 19m
controller-0 0/1 CrashLoopBackOff 6 19m
describe controller 0 ->
Name: controller-0
Namespace: openwhisk
Node: minikube/192.168.86.199
Start Time: Fri, 02 Nov 2018 12:49:30 -0400
Labels: controller-revision-hash=controller-69db44c56b
name=controller
statefulset.kubernetes.io/pod-name=controller-0
Annotations: <none>
Status: Running
IP: 172.17.0.16
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID: docker://13f6ee746d319e0e25c6b33039b5aebf012319ba47b07960cef1298af7e8a937
Image: busybox
Image ID: docker-pullable://busybox@sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 02 Nov 2018 12:49:37 -0400
Finished: Fri, 02 Nov 2018 12:49:56 -0400
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v64m5 (ro)
wait-for-couchdb:
Container ID: docker://e04f0feaf7934db29b9589aec2dfc1a03dd38e7970dd9b8fbea131273ece61b4
Image: busybox
Image ID: docker-pullable://busybox@sha256:2a03a6059f21e150ae84b0973863609494aad70f0a80eaeb64bddd8d92465812
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 02 Nov 2018 12:50:02 -0400
Finished: Fri, 02 Nov 2018 12:53:54 -0400
Ready: True
Restart Count: 0
Environment:
READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v64m5 (ro)
Containers:
controller:
Container ID: docker://2540189bd3a7e7d1d9fda096cc62502603a71fce4f14ac7ff66dd43ae9536dd2
Image: openwhisk/controller:latest
Image ID: docker-pullable://openwhisk/controller@sha256:aa2498cb3c7b7ae08271019c88f7832b4daf28b1da72fe96d8f6cd59e594eba5
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | cut -d'-' -f2`
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 02 Nov 2018 13:07:35 -0400
Finished: Fri, 02 Nov 2018 13:07:41 -0400
Ready: False
Restart Count: 6
Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
PORT: 8080
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092
KAFKA_HOST_PORT: 9092
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-v64m5 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
ow-core-token-v64m5:
Type: Secret (a volume populated by a Secret)
SecretName: ow-core-token-v64m5
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned controller-0 to minikube
Normal SuccessfulMountVolume 20m kubelet, minikube MountVolume.SetUp succeeded for volume "ow-core-token-v64m5"
Normal Pulled 20m kubelet, minikube Container image "busybox" already present on machine
Normal Created 20m kubelet, minikube Created container
Normal Started 20m kubelet, minikube Started container
Normal Pulled 20m kubelet, minikube Container image "busybox" already present on machine
Normal Created 20m kubelet, minikube Created container
Normal Started 20m kubelet, minikube Started container
Warning Unhealthy 14m (x5 over 15m) kubelet, minikube Liveness probe failed: Get http://172.17.0.16:8080/ping: dial tcp 172.17.0.16:8080: getsockopt: connection refused
Normal Pulling 14m (x3 over 16m) kubelet, minikube pulling image "openwhisk/controller:latest"
Normal Killing 14m (x2 over 15m) kubelet, minikube Killing container with id docker://controller:Container failed liveness probe.. Container will be killed and recreated.
Normal Pulled 14m (x3 over 16m) kubelet, minikube Successfully pulled image "openwhisk/controller:latest"
Normal Created 14m (x3 over 16m) kubelet, minikube Created container
Normal Started 14m (x3 over 16m) kubelet, minikube Started container
Warning BackOff 28s (x37 over 8m) kubelet, minikube Back-off restarting failed container
λ kubectl logs pod/controller-0 -n openwhisk
Picked up JAVA_TOOL_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
[2018-11-02T17:07:38.044Z] [INFO] Initializing Kamon...
[INFO] [11/02/2018 17:07:39.043] [main] [StatsDExtension(akka://kamon)] Starting the Kamon(StatsD) extension
[2018-11-02T17:07:39.107Z] [INFO] Slf4jLogger started
[2018-11-02T17:07:39.566Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.triggers.fires.perMinute
[2018-11-02T17:07:39.568Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.sequence.maxLength
[2018-11-02T17:07:39.569Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.concurrent
[2018-11-02T17:07:39.570Z] [INFO] [#tid_sid_unknown] [Config] environment set value for controller.instances
[2018-11-02T17:07:39.571Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.perMinute
[2018-11-02T17:07:39.572Z] [INFO] [#tid_sid_unknown] [Config] environment set value for runtimes.manifest
[2018-11-02T17:07:39.573Z] [INFO] [#tid_sid_unknown] [Config] environment set value for kafka.hosts
[2018-11-02T17:07:39.575Z] [INFO] [#tid_sid_unknown] [Config] environment set value for port
[2018-11-02T17:07:41.071Z] [ERROR] [#tid_sid_unknown] [KafkaMessagingProvider] ensureTopic for completed0 failed due to java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownServerException: The server experienced an unexpected error when processing the request
[2018-11-02T17:07:41.078Z] [ERROR] [#tid_sid_unknown] [Controller] failure during msgProvider.ensureTopic for topic completed0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services