You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2020/01/24 11:43:25 UTC
[GitHub] [openwhisk-deploy-kube] nikolavukovic opened a new issue #577:
Kafka, Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
nikolavukovic opened a new issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577
I have set up a 2 nodes cluster (master and one worker) using Vagrant following technical requirements written here. I am trying to install Openwhisk using Helm 3. Kafka, Controller and Invoker pods are stuck in Init phase and I can't find the problem using the logs. I will put some information that could be useful for you.
```
kubectl get all -n openwhisk -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/owdev-alarmprovider-686b9796f7-f9hp9 0/1 Init:0/1 0 12m 10.32.0.5 worker <none> <none>
pod/owdev-apigateway-6b5686f4c9-dwbl4 1/1 Running 0 12m 10.32.0.14 worker <none> <none>
pod/owdev-cloudantprovider-7445484996-tj2z5 0/1 Init:0/1 0 12m 10.32.0.4 worker <none> <none>
pod/owdev-controller-0 0/1 Init:0/2 0 12m 10.32.0.17 worker <none> <none>
pod/owdev-couchdb-59cc777bbc-jb6gd 1/1 Running 0 12m 10.32.0.13 worker <none> <none>
pod/owdev-gen-certs-4rzzl 0/1 Completed 0 12m 10.32.0.9 worker <none> <none>
pod/owdev-init-couchdb-xlbch 1/1 Running 0 12m 10.32.0.10 worker <none> <none>
pod/owdev-install-packages-7kz5v 0/1 Init:0/1 0 12m 10.32.0.15 worker <none> <none>
pod/owdev-invoker-0 0/1 Init:0/1 0 12m 10.32.0.16 worker <none> <none>
pod/owdev-kafka-0 0/1 Init:0/1 0 12m 10.32.0.6 worker <none> <none>
pod/owdev-kafkaprovider-bf858d897-g4vqt 0/1 Init:0/1 0 12m 10.32.0.8 worker <none> <none>
pod/owdev-nginx-8495fdcb9c-htbb2 1/1 Running 0 12m 10.32.0.18 worker <none> <none>
pod/owdev-redis-75674fd487-nzns7 1/1 Running 0 12m 10.32.0.11 worker <none> <none>
pod/owdev-wskadmin 1/1 Running 0 12m 10.32.0.3 worker <none> <none>
pod/owdev-zookeeper-0 1/1 Running 0 12m 10.32.0.12 worker <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/owdev-apigateway ClusterIP 10.106.233.102 <none> 8080/TCP,9000/TCP 12m name=owdev-apigateway
service/owdev-controller ClusterIP 10.98.99.59 <none> 8080/TCP 12m name=owdev-controller
service/owdev-couchdb ClusterIP 10.108.235.142 <none> 5984/TCP 12m name=owdev-couchdb
service/owdev-kafka ClusterIP None <none> 9092/TCP 12m name=owdev-kafka
service/owdev-nginx NodePort 10.106.198.121 <none> 80:32511/TCP,443:31001/TCP 12m name=owdev-nginx
service/owdev-redis ClusterIP 10.100.249.168 <none> 6379/TCP 12m name=owdev-redis
service/owdev-zookeeper ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 12m name=owdev-zookeeper
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/owdev-alarmprovider 0/1 1 0 12m alarmprovider openwhisk/alarmprovider:2.0.0-incubating name=owdev-alarmprovider
deployment.apps/owdev-apigateway 1/1 1 1 12m apigateway openwhisk/apigateway:0.11.0 name=owdev-apigateway
deployment.apps/owdev-cloudantprovider 0/1 1 0 12m cloudantprovider openwhisk/cloudantprovider:2.0.0-incubating name=owdev-cloudantprovider
deployment.apps/owdev-couchdb 1/1 1 1 12m couchdb apache/couchdb:2.3 name=owdev-couchdb
deployment.apps/owdev-kafkaprovider 0/1 1 0 12m kafkaprovider openwhisk/kafkaprovider:2.0.0-incubating name=owdev-kafkaprovider
deployment.apps/owdev-nginx 1/1 1 1 12m nginx nginx:1.17 name=owdev-nginx
deployment.apps/owdev-redis 1/1 1 1 12m redis redis:4.0 name=owdev-redis
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/owdev-alarmprovider-686b9796f7 1 1 0 12m alarmprovider openwhisk/alarmprovider:2.0.0-incubating name=owdev-alarmprovider,pod-template-hash=686b9796f7replicaset.apps/owdev-apigateway-6b5686f4c9 1 1 1 12m apigateway openwhisk/apigateway:0.11.0 name=owdev-apigateway,pod-template-hash=6b5686f4c9 replicaset.apps/owdev-cloudantprovider-7445484996 1 1 0 12m cloudantprovider openwhisk/cloudantprovider:2.0.0-incubating name=owdev-cloudantprovider,pod-template-hash=7445484996
replicaset.apps/owdev-couchdb-59cc777bbc 1 1 1 12m couchdb apache/couchdb:2.3 name=owdev-couchdb,pod-template-hash=59cc777bbc replicaset.apps/owdev-kafkaprovider-bf858d897 1 1 0 12m kafkaprovider openwhisk/kafkaprovider:2.0.0-incubating name=owdev-kafkaprovider,pod-template-hash=bf858d897 replicaset.apps/owdev-nginx-8495fdcb9c 1 1 1 12m nginx nginx:1.17 name=owdev-nginx,pod-template-hash=8495fdcb9c replicaset.apps/owdev-redis-75674fd487 1 1 1 12m redis redis:4.0 name=owdev-redis,pod-template-hash=75674fd487
NAME READY AGE CONTAINERS IMAGES
statefulset.apps/owdev-controller 0/1 12m controller openwhisk/controller:8eb922f
statefulset.apps/owdev-invoker 0/1 12m invoker openwhisk/invoker:8eb922f
statefulset.apps/owdev-kafka 0/1 12m kafka wurstmeister/kafka:0.11.0.1
statefulset.apps/owdev-zookeeper 1/1 12m zookeeper zookeeper:3.4
NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR
job.batch/owdev-gen-certs 1/1 12s 12m gen-certs openwhisk/ow-utils:8eb922f controller-uid=d09ee120-728a-4e9a-8922-904639f7f1dd
job.batch/owdev-init-couchdb 0/1 12m 12m init-couchdb openwhisk/ow-utils:8eb922f controller-uid=27c47d17-6701-4254-b4b6-dd0b90acbaba
job.batch/owdev-install-packages 0/1 12m 12m install-packages openwhisk/ow-utils:8eb922f controller-uid=58aaddb9-2016-4499-9706-7ec4ff7922ad
```
Shortened version of invoker
```
kubectl describe pod -n openwhisk owdev-invoker-0
Name: owdev-invoker-0
Namespace: openwhisk
Priority: 0
Node: worker/172.42.42.101
Start Time: Fri, 24 Jan 2020 10:59:50 +0000
Labels: app=owdev-openwhisk
chart=openwhisk-0.1.10
controller-revision-hash=owdev-invoker-6c8967fdb5
heritage=Helm
name=owdev-invoker
release=owdev
statefulset.kubernetes.io/pod-name=owdev-invoker-0
Annotations: <none>
Status: Pending
IP: 10.32.0.16
IPs:
IP: 10.32.0.16
Controlled By: StatefulSet/owdev-invoker
Init Containers:
wait-for-controller:
Container ID: docker://72979b7d13ea6e2e34dea3d33c79165e1e9d328bc1b9ecb321afba99ff2bf8b8
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'
State: Running
Started: Fri, 24 Jan 2020 10:59:57 +0000
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://owdev-controller.openwhisk.svc.kubernetes.default.svc.cluster.local:8080/ping
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-invoker-token-kjv8m (ro)
Containers:
invoker:
Container ID:
Image: openwhisk/invoker:8eb922f
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
/init.sh --uniqueName $INVOKER_NAME
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PORT: 8080
WHISK_API_HOST_PROTO: <set to the key 'whisk_api_host_proto' of config map 'owdev-whisk.config'> Optional: false
WHISK_API_HOST_PORT: <set to the key 'whisk_api_host_port' of config map 'owdev-whisk.config'> Optional: false
WHISK_API_HOST_NAME: <set to the key 'whisk_api_host_name' of config map 'owdev-whisk.config'> Optional: false
CONFIG_whisk_docker_containerFactory_useRunc: false
CONFIG_whisk_containerPool_userMemory: 2048m
CONFIG_whisk_containerFactory_containerArgs_network: bridge
CONFIG_whisk_containerFactory_containerArgs_extraEnvVars_0: __OW_ALLOW_CONCURRENT=false
CONFIG_whisk_containerFactory_runtimesRegistry_url:
INVOKER_NAME: owdev-invoker-0 (v1:metadata.name)
JAVA_OPTS: -Xmx512M
INVOKER_OPTS: -Dkubernetes.master=https://$KUBERNETES_SERVICE_HOST -Dwhisk.spi.ContainerFactoryProvider=org.apache.openwhisk.core.containerpool.kubernetes.KubernetesContainerFactoryProvider
RUNTIMES_MANIFEST: {
....................................
ZOOKEEPER_HOSTS: owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local:2181
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-invoker-token-kjv8m (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
owdev-invoker-token-kjv8m:
Type: Secret (a volume populated by a Secret)
SecretName: owdev-invoker-token-kjv8m
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
openwhisk-role=invoker:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned openwhisk/owdev-invoker-0 to worker
Normal Pulled 23m kubelet, worker Container image "busybox" already present on machine
Normal Created 23m kubelet, worker Created container wait-for-controller
Normal Started 23m kubelet, worker Started container wait-for-controller
```
Shortened version of controller
```
kubectl describe pod -n openwhisk owdev-controller-0
Name: owdev-controller-0
Namespace: openwhisk
Priority: 0
Node: worker/172.42.42.101
Start Time: Fri, 24 Jan 2020 10:59:50 +0000
Labels: app=owdev-openwhisk
chart=openwhisk-0.1.10
controller-revision-hash=owdev-controller-65c995d96c
heritage=Helm
name=owdev-controller
release=owdev
statefulset.kubernetes.io/pod-name=owdev-controller-0
Annotations: <none>
Status: Pending
IP: 10.32.0.17
IPs:
IP: 10.32.0.17
Controlled By: StatefulSet/owdev-controller
Init Containers:
wait-for-kafka:
Container ID: docker://a1bde85e6c8d402203964f9e65be57ccd5c1714c8b1c5e26022552e252b91442
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
State: Running
Started: Fri, 24 Jan 2020 10:59:57 +0000
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro)
wait-for-couchdb:
Container ID:
Image: busybox
Image ID:
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://owdev-couchdb.openwhisk.svc.kubernetes.default.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro)
Containers:
controller:
Container ID:
Image: openwhisk/controller:8eb922f
Image ID:
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | awk -F '-' '{print $NF}'`
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8080/ping delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/ping delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
PORT: 8080
TZ: UTC
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'owdev-whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'owdev-whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
........................................................
CONFIG_whisk_kafka_topics_events_segmentBytes:
CONFIG_whisk_kafka_topics_health_retentionBytes:
CONFIG_whisk_kafka_topics_health_retentionMs:
CONFIG_whisk_kafka_topics_health_segmentBytes:
CONFIG_whisk_kafka_topics_invoker_retentionBytes:
CONFIG_whisk_kafka_topics_invoker_retentionMs:
CONFIG_whisk_kafka_topics_invoker_segmentBytes:
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'owdev-db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'owdev-db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_host: owdev-couchdb.openwhisk.svc.kubernetes.default.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'owdev-db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'owdev-db.config'> Optional: false
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-2qh55 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
owdev-core-token-2qh55:
Type: Secret (a volume populated by a Secret)
SecretName: owdev-core-token-2qh55
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
openwhisk-role=core:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26m default-scheduler Successfully assigned openwhisk/owdev-controller-0 to worker
Warning FailedMount 26m kubelet, worker MountVolume.SetUp failed for volume "owdev-core-token-2qh55" : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 26m kubelet, worker Container image "busybox" already present on machine
Normal Created 26m kubelet, worker Created container wait-for-kafka
Normal Started 26m kubelet, worker Started container wait-for-kafka
```
```
kubectl describe pod -n openwhisk owdev-kafka-0
Name: owdev-kafka-0
Namespace: openwhisk
Priority: 0
Node: worker/172.42.42.101
Start Time: Fri, 24 Jan 2020 10:59:50 +0000
Labels: app=owdev-openwhisk
chart=openwhisk-0.1.10
controller-revision-hash=owdev-kafka-847cd555bf
heritage=Helm
name=owdev-kafka
release=owdev
statefulset.kubernetes.io/pod-name=owdev-kafka-0
Annotations: <none>
Status: Pending
IP: 10.32.0.6
IPs:
IP: 10.32.0.6
Controlled By: StatefulSet/owdev-kafka
Init Containers:
wait-for-zookeeper:
Container ID: docker://4b9db4e302ddf91200721d537b150c4b28b5942c4a4a71ea9adfe5eba2162881
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up"
State: Running
Started: Fri, 24 Jan 2020 10:59:56 +0000
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
Containers:
kafka:
Container ID:
Image: wurstmeister/kafka:0.11.0.1
Image ID:
Port: 9092/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
export KAFKA_BROKER_ID=`hostname | awk -F '-' '{print $NF}'`; export KAFKA_ADVERTISED_HOST_NAME=`hostname -f`; /usr/bin/start-kafka.sh
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local:2181
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-bjt6q:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bjt6q
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
openwhisk-role=core:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned openwhisk/owdev-kafka-0 to worker
Normal Pulled 27m kubelet, worker Container image "busybox" already present on machine
Normal Created 27m kubelet, worker Created container wait-for-zookeeper
Normal Started 27m kubelet, worker Started container wait-for-zookeeper
```
```
kubectl describe pod -n openwhisk owdev-kafkaprovider-bf858d897-g4vqt
Name: owdev-kafkaprovider-bf858d897-g4vqt
Namespace: openwhisk
Priority: 0
Node: worker/172.42.42.101
Start Time: Fri, 24 Jan 2020 10:59:50 +0000
Labels: app=owdev-openwhisk
chart=openwhisk-0.1.10
heritage=Helm
name=owdev-kafkaprovider
pod-template-hash=bf858d897
release=owdev
Annotations: <none>
Status: Pending
IP: 10.32.0.8
IPs:
IP: 10.32.0.8
Controlled By: ReplicaSet/owdev-kafkaprovider-bf858d897
Init Containers:
wait-for-controller:
Container ID: docker://6139e487e23f903f1275f60e9296fc42c763918c2cb14d48d1ddc713f8a4d38c
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'
State: Running
Started: Fri, 24 Jan 2020 10:59:56 +0000
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://owdev-controller.openwhisk.svc.kubernetes.default.svc.cluster.local:8080/ping
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
Containers:
kafkaprovider:
Container ID:
Image: openwhisk/kafkaprovider:2.0.0-incubating
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
DB_URL: <set to the key 'db_url' of config map 'owdev-db.config'> Optional: false
DB_USER: <set to the key 'db_username' in secret 'owdev-db.auth'> Optional: false
DB_PASS: <set to the key 'db_password' in secret 'owdev-db.auth'> Optional: false
DB_PREFIX: kp
LOCAL_DEV: true
ROUTER_HOST: <set to the key 'whisk_api_host_nameAndPort' of config map 'owdev-whisk.config'> Optional: false
ENDPOINT_AUTH: <set to the key 'whisk_api_host_nameAndPort' of config map 'owdev-whisk.config'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-bjt6q:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bjt6q
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28m default-scheduler Successfully assigned openwhisk/owdev-kafkaprovider-bf858d897-g4vqt to worker
Normal Pulled 28m kubelet, worker Container image "busybox" already present on machine
Normal Created 28m kubelet, worker Created container wait-for-controller
Normal Started 28m kubelet, worker Started container wait-for-controller
```
```
kubectl describe pod -n openwhisk owdev-zookeeper-0
Name: owdev-zookeeper-0
Namespace: openwhisk
Priority: 0
Node: worker/172.42.42.101
Start Time: Fri, 24 Jan 2020 10:59:50 +0000
Labels: app=owdev-openwhisk
chart=openwhisk-0.1.10
controller-revision-hash=owdev-zookeeper-6bc7d5cf48
heritage=Helm
name=owdev-zookeeper
release=owdev
statefulset.kubernetes.io/pod-name=owdev-zookeeper-0
Annotations: <none>
Status: Running
IP: 10.32.0.12
IPs:
IP: 10.32.0.12
Controlled By: StatefulSet/owdev-zookeeper
Containers:
zookeeper:
Container ID: docker://93ee6f00abab5ec6bc16a78c6c9808a87d1a33f5ffb9b9f1bcca393236432c49
Image: zookeeper:3.4
Image ID: docker-pullable://zookeeper@sha256:491427fc9f788c168e096422afe620fdb269a5d604efd11f953682919101c658
Ports: 2181/TCP, 2888/TCP, 3888/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
hostname -s | awk -F '-' '{print $NF}'> /data/myid; cat /data/myid; cat /conf/zoo.cfg; zkServer.sh start-foreground
State: Running
Started: Fri, 24 Jan 2020 10:59:57 +0000
Ready: True
Restart Count: 0
Liveness: tcp-socket :2181 delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -c echo ruok | nc -w 1 localhost 2181 | grep imok] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/conf from zk-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
zk-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: owdev-zookeeper
Optional: false
default-token-bjt6q:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bjt6q
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
openwhisk-role=core:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 29m default-scheduler Successfully assigned openwhisk/owdev-zookeeper-0 to worker
Normal Pulled 29m kubelet, worker Container image "zookeeper:3.4" already present on machine
Normal Created 29m kubelet, worker Created container zookeeper
Normal Started 29m kubelet, worker Started container zookeeper
```
What I understand from this is that invoker is waiting for controller, controller is waiting for kafka, kafka is waiting for zookeeper. There are docker containers running on worker executing commands printed here.
There is another problem with couchdb pods. They are running but the DB is not correctly initialized. Following are the logs
```
kubectl logs -n openwhisk owdev-init-couchdb-xlbch
Cloning into '/openwhisk'...
/openwhisk /
Note: checking out '8eb922f568d7843cdb1068252d29ba5723b30283'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 8eb922f5 Separate test cases for the logs and results APIs of activation (#4777)
/
/openwhisk/ansible /
[WARNING]: Unable to parse /openwhisk/ansible/environments/local as an
inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note
that the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [Gathering Facts] *********************************************************
Friday 24 January 2020 11:00:44 +0000 (0:00:00.275) 0:00:00.275 ********
ok: [localhost]
TASK [gen hosts if 'local' env is used] ****************************************
Friday 24 January 2020 11:00:46 +0000 (0:00:01.673) 0:00:01.950 ********
changed: [localhost -> localhost]
TASK [find the ip of docker-machine] *******************************************
Friday 24 January 2020 11:00:46 +0000 (0:00:00.865) 0:00:02.816 ********
skipping: [localhost]
TASK [get the docker-machine ip] ***********************************************
Friday 24 January 2020 11:00:47 +0000 (0:00:00.043) 0:00:02.860 ********
skipping: [localhost]
TASK [gen hosts for docker-machine] ********************************************
Friday 24 January 2020 11:00:47 +0000 (0:00:00.060) 0:00:02.921 ********
skipping: [localhost]
TASK [gen hosts for Jenkins] ***************************************************
Friday 24 January 2020 11:00:47 +0000 (0:00:00.059) 0:00:02.980 ********
skipping: [localhost]
TASK [check if db_local.ini exists?] *******************************************
Friday 24 January 2020 11:00:47 +0000 (0:00:00.054) 0:00:03.035 ********
ok: [localhost]
TASK [prepare db_local.ini] ****************************************************
Friday 24 January 2020 11:00:47 +0000 (0:00:00.335) 0:00:03.371 ********
changed: [localhost -> localhost]
TASK [gen untrusted server certificate for host] *******************************
Friday 24 January 2020 11:00:48 +0000 (0:00:00.512) 0:00:03.884 ********
changed: [localhost -> localhost]
TASK [gen untrusted client certificate for host] *******************************
Friday 24 January 2020 11:00:49 +0000 (0:00:00.957) 0:00:04.842 ********
changed: [localhost -> localhost]
TASK [clean up old kafka keystore] *********************************************
Friday 24 January 2020 11:00:49 +0000 (0:00:00.965) 0:00:05.808 ********
skipping: [localhost]
TASK [ensure kafka files directory exists] *************************************
Friday 24 January 2020 11:00:50 +0000 (0:00:00.052) 0:00:05.861 ********
skipping: [localhost]
TASK [generate kafka certificates] *********************************************
Friday 24 January 2020 11:00:50 +0000 (0:00:00.078) 0:00:05.940 ********
skipping: [localhost]
TASK [ensure controller files directory exists] ********************************
Friday 24 January 2020 11:00:50 +0000 (0:00:00.047) 0:00:05.987 ********
changed: [localhost]
TASK [generate controller certificates] ****************************************
Friday 24 January 2020 11:00:51 +0000 (0:00:00.975) 0:00:06.963 ********
changed: [localhost -> localhost]
TASK [ensure invoker files directory exists] ***********************************
Friday 24 January 2020 11:00:52 +0000 (0:00:01.161) 0:00:08.125 ********
changed: [localhost]
TASK [generate invoker certificates] *******************************************
Friday 24 January 2020 11:00:52 +0000 (0:00:00.692) 0:00:08.818 ********
changed: [localhost -> localhost]
PLAY RECAP *********************************************************************
localhost : ok=10 changed=8 unreachable=0 failed=0
Friday 24 January 2020 11:00:54 +0000 (0:00:01.083) 0:00:09.901 ********
===============================================================================
Gathering Facts --------------------------------------------------------- 1.68s
generate controller certificates ---------------------------------------- 1.16s
generate invoker certificates ------------------------------------------- 1.08s
ensure controller files directory exists -------------------------------- 0.98s
gen untrusted client certificate for host ------------------------------- 0.97s
gen untrusted server certificate for host ------------------------------- 0.96s
gen hosts if 'local' env is used ---------------------------------------- 0.87s
ensure invoker files directory exists ----------------------------------- 0.69s
prepare db_local.ini ---------------------------------------------------- 0.51s
check if db_local.ini exists? ------------------------------------------- 0.34s
ensure kafka files directory exists ------------------------------------- 0.08s
get the docker-machine ip ----------------------------------------------- 0.06s
gen hosts for docker-machine -------------------------------------------- 0.06s
gen hosts for Jenkins --------------------------------------------------- 0.06s
clean up old kafka keystore --------------------------------------------- 0.05s
generate kafka certificates --------------------------------------------- 0.05s
find the ip of docker-machine ------------------------------------------- 0.04s
PLAY [db] **********************************************************************
TASK [Gathering Facts] *********************************************************
Friday 24 January 2020 11:00:56 +0000 (0:00:00.269) 0:00:00.269 ********
ok: [172.17.0.1]
PLAY RECAP *********************************************************************
172.17.0.1 : ok=1 changed=0 unreachable=0 failed=0
Friday 24 January 2020 11:00:58 +0000 (0:00:01.591) 0:00:01.861 ********
===============================================================================
Gathering Facts --------------------------------------------------------- 1.59s
/
waiting for CouchDB to be available
waiting for CouchDB to be available
waiting for CouchDB to be available
waiting for CouchDB to be available
waiting for CouchDB to be available
waiting for CouchDB to be available
waiting for CouchDB to be available
...
```
```
kubectl logs -n openwhisk owdev-couchdb-59cc777bbc-jb6gd
[info] 2020-01-24T11:00:00.198305Z couchdb@couchdb0 <0.9.0> -------- Application couch_log started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.213370Z couchdb@couchdb0 <0.9.0> -------- Application folsom started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.388557Z couchdb@couchdb0 <0.9.0> -------- Application couch_stats started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.389234Z couchdb@couchdb0 <0.9.0> -------- Application khash started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.412502Z couchdb@couchdb0 <0.9.0> -------- Application couch_event started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.413146Z couchdb@couchdb0 <0.9.0> -------- Application hyper started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.430790Z couchdb@couchdb0 <0.9.0> -------- Application ibrowse started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.453061Z couchdb@couchdb0 <0.9.0> -------- Application ioq started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.454319Z couchdb@couchdb0 <0.9.0> -------- Application mochiweb started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.491742Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB 2.3.1 is starting.
[info] 2020-01-24T11:00:00.491926Z couchdb@couchdb0 <0.213.0> -------- Starting couch_sup
[notice] 2020-01-24T11:00:00.541999Z couchdb@couchdb0 <0.96.0> -------- config: [features] pluggable-storage-engines set to true for reason nil
[notice] 2020-01-24T11:00:00.566597Z couchdb@couchdb0 <0.96.0> -------- config: [admins] whisk_admin set to -pbkdf2-3a1c30b3e9965585f6d0cd9ff907ede232f0847f,2f06a1ea99722b3696d260d440f791c4,10 for reason nil
[notice] 2020-01-24T11:00:00.654765Z couchdb@couchdb0 <0.96.0> -------- config: [couchdb] uuid set to 3a9c7e0f5a9b50a69616239d1619cc6d for reason nil
[info] 2020-01-24T11:00:00.779773Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _users
[info] 2020-01-24T11:00:00.933696Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB has started. Time to relax.
[info] 2020-01-24T11:00:00.933868Z couchdb@couchdb0 <0.212.0> -------- Apache CouchDB has started on http://any:5986/
[info] 2020-01-24T11:00:00.934031Z couchdb@couchdb0 <0.9.0> -------- Application couch started on node couchdb@couchdb0
[info] 2020-01-24T11:00:00.934156Z couchdb@couchdb0 <0.9.0> -------- Application ets_lru started on node couchdb@couchdb0
[notice] 2020-01-24T11:00:01.028698Z couchdb@couchdb0 <0.285.0> -------- rexi_server : started servers
[notice] 2020-01-24T11:00:01.039627Z couchdb@couchdb0 <0.289.0> -------- rexi_buffer : started servers
[info] 2020-01-24T11:00:01.040398Z couchdb@couchdb0 <0.9.0> -------- Application rexi started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.060380Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _nodes
[warning] 2020-01-24T11:00:01.060443Z couchdb@couchdb0 <0.297.0> -------- creating missing database: _nodes
[info] 2020-01-24T11:00:01.156452Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _dbs
[warning] 2020-01-24T11:00:01.156534Z couchdb@couchdb0 <0.311.0> -------- creating missing database: _dbs
[info] 2020-01-24T11:00:01.165833Z couchdb@couchdb0 <0.9.0> -------- Application mem3 started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.166078Z couchdb@couchdb0 <0.9.0> -------- Application fabric started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.202113Z couchdb@couchdb0 <0.9.0> -------- Application chttpd started on node couchdb@couchdb0
[error] 2020-01-24T11:00:01.226620Z couchdb@couchdb0 emulator -------- Error in process <0.351.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
[notice] 2020-01-24T11:00:01.226735Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[info] 2020-01-24T11:00:01.238657Z couchdb@couchdb0 <0.9.0> -------- Application couch_index started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.238715Z couchdb@couchdb0 <0.9.0> -------- Application couch_mrview started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.238775Z couchdb@couchdb0 <0.9.0> -------- Application couch_plugins started on node couchdb@couchdb0
[notice] 2020-01-24T11:00:01.287124Z couchdb@couchdb0 <0.96.0> -------- config: [features] scheduler set to true for reason nil
[info] 2020-01-24T11:00:01.316012Z couchdb@couchdb0 <0.218.0> -------- open_result error {not_found,no_db_file} for _replicator
[notice] 2020-01-24T11:00:01.319811Z couchdb@couchdb0 <0.368.0> -------- creating replicator ddoc <<"_replicator">>
[info] 2020-01-24T11:00:01.331007Z couchdb@couchdb0 <0.9.0> -------- Application couch_replicator started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.350148Z couchdb@couchdb0 <0.9.0> -------- Application couch_peruser started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.374999Z couchdb@couchdb0 <0.9.0> -------- Application ddoc_cache started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.397739Z couchdb@couchdb0 <0.9.0> -------- Application global_changes started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.397811Z couchdb@couchdb0 <0.9.0> -------- Application jiffy started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.407443Z couchdb@couchdb0 <0.9.0> -------- Application mango started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.416278Z couchdb@couchdb0 <0.9.0> -------- Application setup started on node couchdb@couchdb0
[info] 2020-01-24T11:00:01.416325Z couchdb@couchdb0 <0.9.0> -------- Application snappy started on node couchdb@couchdb0
[notice] 2020-01-24T11:00:06.227704Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[error] 2020-01-24T11:00:06.228333Z couchdb@couchdb0 emulator -------- Error in process <0.490.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
[notice] 2020-01-24T11:00:06.254012Z couchdb@couchdb0 <0.372.0> -------- couch_replicator_clustering : cluster stable
[notice] 2020-01-24T11:00:06.261084Z couchdb@couchdb0 <0.389.0> -------- Started replicator db changes listener <0.491.0>
[error] 2020-01-24T11:00:11.228985Z couchdb@couchdb0 emulator -------- Error in process <0.590.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
[notice] 2020-01-24T11:00:11.229059Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[notice] 2020-01-24T11:00:16.029104Z couchdb@couchdb0 <0.289.0> -------- rexi_buffer : cluster stable
[notice] 2020-01-24T11:00:16.029228Z couchdb@couchdb0 <0.285.0> -------- rexi_server : cluster stable
[error] 2020-01-24T11:00:16.231300Z couchdb@couchdb0 emulator -------- Error in process <0.655.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
[notice] 2020-01-24T11:00:16.231504Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[error] 2020-01-24T11:00:21.232975Z couchdb@couchdb0 emulator -------- Error in process <0.752.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
[notice] 2020-01-24T11:00:21.233053Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[notice] 2020-01-24T11:00:26.235354Z couchdb@couchdb0 <0.350.0> -------- chttpd_auth_cache changes listener died database_does_not_exist at mem3_shards:load_shards_from_db/6(line:395) <= mem3_shards:load_shards_from_disk/1(line:370) <= mem3_shards:load_shards_from_disk/2(line:399) <= mem3_shards:for_docid/3(line:86) <= fabric_doc_open:go/3(line:39) <= chttpd_auth_cache:ensure_auth_ddoc_exists/2(line:195) <= chttpd_auth_cache:listen_for_changes/1(line:142)
[error] 2020-01-24T11:00:26.235514Z couchdb@couchdb0 emulator -------- Error in process <0.817.0> on node couchdb@couchdb0 with exit value:
{database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,395}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,86}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,195}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,142}]}]}
...
```
I tried setting an external CouchDB and putting new values in mycluster.yaml to redirect Openwhisk to it. CouchDB pods were not initialized and I guess that is a sign of normal execution.
Anyway I would like to solve this problem with DB and with other pods. Can you please help? If you need any other logs just tell.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] nikolavukovic closed issue #577: Kafka,
Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
nikolavukovic closed issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] dgrove-oss commented on issue #577: Kafka,
Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
dgrove-oss commented on issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577#issuecomment-578178355
Looks to me like you have most likely some kind of networking problem. The kafka pod's init_container doesn't think zookeeper is ready.
````
Init Containers:
wait-for-zookeeper:
Container ID: docker://4b9db4e302ddf91200721d537b150c4b28b5942c4a4a71ea9adfe5eba2162881
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up"
State: Running
Started: Fri, 24 Jan 2020 10:59:56 +0000
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
```
Either zookeeper really isn't ready (in which case debug why zookeeper isn't starting correctly) or the probe in the can't reach zookeeper to get the expected `imok` response (so debug why the network connectivity isn't as expected).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] nikolavukovic commented on issue #577:
Kafka, Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
nikolavukovic commented on issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577#issuecomment-579392440
I am not sure what exactly happened, but it is working now that I have deleted this line
```
k8s:
domain: kubernetes.default.svc.cluster.local
```
from mycluster.yaml. The domain is now set to cluster.local as predefined.
Also, I was using Flannel as CNI plugin but now I am using Weave.
Thanks for suggesting that it was a networking problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] dgrove-oss commented on issue #577: Kafka,
Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
dgrove-oss commented on issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577#issuecomment-578465334
Sorry, I've never tried to set up a Kubernetes cluster on top of Vagrant before. It looks to me like your setup has networking problems that could be debugged using a much simpler system than openwhisk (eg just two pods + services that try to talk to each other). I'd suggest trying to get that working first, then get back to putting OpenWhisk on the cluster once the basics are working.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] dgrove-oss edited a comment on issue #577:
Kafka, Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
dgrove-oss edited a comment on issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577#issuecomment-578178355
Looks to me like you have most likely some kind of networking problem. The kafka pod's init_container doesn't think zookeeper is ready. Either zookeeper really isn't ready (in which case debug why zookeeper isn't starting correctly) or the probe can't reach zookeeper to get the expected `imok` response (so debug why the network connectivity isn't as expected).
````
Init Containers:
wait-for-zookeeper:
Container ID: docker://4b9db4e302ddf91200721d537b150c4b28b5942c4a4a71ea9adfe5eba2162881
Image: busybox
Image ID: docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up"
State: Running
Started: Fri, 24 Jan 2020 10:59:56 +0000
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-bjt6q (ro)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk-deploy-kube] nikolavukovic commented on issue #577:
Kafka, Controller and Invoker pods stuck in Init phase,
CouchDB not initialized properly
Posted by GitBox <gi...@apache.org>.
nikolavukovic commented on issue #577: Kafka, Controller and Invoker pods stuck in Init phase, CouchDB not initialized properly
URL: https://github.com/apache/openwhisk-deploy-kube/issues/577#issuecomment-578306002
Hi David, many thanks for replying!
Indeed, zookeeper is not reachable from the kafka pod container. I entered the container and tried to executed the command. This is what I get:
```
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up"
nc: bad address 'owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local'
waiting for zookeeper to be ready
nc: bad address 'owdev-zookeeper-0.owdev-zookeeper.openwhisk.svc.kubernetes.default.svc.cluster.local'
```
Another problem I had before this is that nginx pod would enter into CrashLoopBackOff state because it had a networking problem. In mycluster.yaml I had to put
```
k8s:
domain: kubernetes.default.svc.cluster.local
```
instead of
```
k8s:
domain: cluster.local
```
I was following another issue started here before related to networking and so I changed domain value without really being aware of the consequences. Can you explain what could be done in this case and why cluster.local was not working?
Basically, this is what I get when using nslookup from nginx pod:
```
root@owdev-nginx-8495fdcb9c-88zhz:/# nslookup cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
*** Can't find cluster.local: No answer
root@owdev-nginx-8495fdcb9c-88zhz:/# nslookup kubernetes.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services