You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2018/10/08 11:30:26 UTC
[GitHub] saurabh1284 opened a new issue #311: controller and invoker pods
are stuck in PodInitializing
saurabh1284 opened a new issue #311: controller and invoker pods are stuck in PodInitializing
URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/311
I have an kubernetes cluster of 2 nodes (1 master and 1 worker) and was trying to deploy openwhisk following documentation steps. But controller and invoker pods are stuck in PodInitializing phase. I am using latest clone.
Please find details as below:
-> kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
server-f2-02.ipc3dev.com Ready master 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-02.ipc3dev.com,node-role.kubernetes.io/master=master
server-f2-0f.ipc3dev.com Ready worker 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-0f.ipc3dev.com,node-role.kubernetes.io/worker=worker,openwhisk-role=invoker
-> kubectl describe nodes | grep InternalIP
nternalIP: 10.71.3.7
InternalIP: 10.71.3.8
-> cat mycluster.yaml
whisk:
ingress:
type: NodePort
api_host_name: 10.71.3.7
api_host_port: 31001
nginx:
httpsNodePort: 31001
-> kubectl -n openwhisk get pods
NAME READY STATUS RESTARTS AGE
apigateway-64b5fb47df-h8v5f 1/1 Running 0 2h
controller-0 0/1 Init:1/2 0 2h
couchdb-6db7d989dd-886mc 1/1 Running 0 2h
invoker-b5mpt 0/1 Init:1/2 0 2h
kafka-0 1/1 Running 0 2h
nginx-55d45778cb-5vgd6 1/1 Running 0 2h
redis-6788bb4875-lcwgw 1/1 Running 0 2h
zookeeper-0 1/1 Running 0 2h
-> kubectl -n openwhisk describe pod controller-0
Name: controller-0
Namespace: openwhisk
Node: server-f2-0f.ipc3dev.com/10.71.3.8
Start Time: Mon, 08 Oct 2018 04:42:25 -0400
Labels: controller-revision-hash=controller-5bc6758f68
name=controller
statefulset.kubernetes.io/pod-name=controller-0
Annotations: <none>
Status: Pending
IP: 10.244.1.156
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID: docker://6d870e1e10ab5913fd8f48c00f7ee342c7e8e9e0af25a727a15d2229ccb4d00a
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:42:37 -0400
Finished: Mon, 08 Oct 2018 04:43:17 -0400
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)
wait-for-couchdb:
Container ID: docker://08391dc6b3eb76127050c35f0948006a0e398d7b761dc208f83c2b621f4a5c64
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Running
Started: Mon, 08 Oct 2018 04:43:19 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)
Containers:
controller:
Container ID:
Image: openwhisk/controller:latest
Image ID:
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | cut -d'-' -f2`
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
PORT: 8080
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092
KAFKA_HOST_PORT: 9092
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
CONTROLLER_INSTANCES: 1
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
ow-core-token-9qfsx:
Type: Secret (a volume populated by a Secret)
SecretName: ow-core-token-9qfsx
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
-> kubectl logs controller-0 -n openwhisk
Error from server (BadRequest): container "controller" in pod "controller-0" is waiting to start: PodInitializing
-> kubectl -n openwhisk describe pod invoker-b5mpt
Name: invoker-b5mpt
Namespace: openwhisk
Node: server-f2-0f.ipc3dev.com/10.71.3.8
Start Time: Mon, 08 Oct 2018 04:42:25 -0400
Labels: controller-revision-hash=137522596
name=invoker
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 10.244.1.154
Controlled By: DaemonSet/invoker
Init Containers:
docker-pull-runtimes:
Container ID: docker://83cffcd527aea83fcf4f9f1849cea44c9bc3fe017e9eb73d932032dc77e27a25
Image: openwhisk/ansible-runner:latest
Image ID: docker-pullable://docker.io/openwhisk/ansible-runner@sha256:5c9a2354b872a86ac08cba8fee79ba7452c55377f26761bc156247b9026d20d6
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:47:05 -0400
Finished: Mon, 08 Oct 2018 05:07:13 -0400
Ready: True
Restart Count: 0
Environment:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
Mounts:
/task/playbook.yml from task-dir (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)
wait-for-controller:
Container ID: docker://b05867e5c3d1632bbbfa79339f379e43fadc6619e690d24d34fd269fa5de922f
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port: <none>
Host Port: <none>
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'
State: Running
Started: Mon, 08 Oct 2018 05:07:14 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://controller.openwhisk.svc.cluster.local:8080/ping
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)
Containers:
invoker:
Container ID:
Image: openwhisk/invoker:latest
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
/init.sh --uniqueName $INVOKER_NAME
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PORT: 8080
WHISK_API_HOST_PROTO: <set to the key 'whisk_api_host_proto' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_PORT: <set to the key 'whisk_api_host_port' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_NAME: <set to the key 'whisk_api_host_name' of config map 'whisk.config'> Optional: false
INVOKER_CONTAINER_NETWORK: bridge
CONFIG_whisk_docker_containerFactory_useRunc: false
DOCKER_IMAGE_PREFIX: openwhisk
DOCKER_IMAGE_TAG: latest
DOCKER_REGISTRY:
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
INVOKER_NAME: (v1:spec.nodeName)
JAVA_OPTS: -Xmx512M
INVOKER_OPTS: -Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.docker.DockerContainerFactoryProvider
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
WHISK_LOGS_DIR:
CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false
CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false
CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false
KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092
ZOOKEEPER_HOSTS: zookeeper-0.zookeeper.openwhisk.svc.cluster.local:2181
Mounts:
/containers from dockerrootdir (rw)
/run/runc from runc (rw)
/sys/fs/cgroup from cgroup (rw)
/var/run/docker.sock from dockersock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cgroup:
Type: HostPath (bare host directory volume)
Path: /sys/fs/cgroup
HostPathType:
runc:
Type: HostPath (bare host directory volume)
Path: /run/runc
HostPathType:
dockerrootdir:
Type: HostPath (bare host directory volume)
Path: /var/lib/docker/containers
HostPathType:
dockersock:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
task-dir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: docker-pull-runtimes
Optional: false
default-token-m4d8z:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-m4d8z
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
-> kubectl logs invoker-b5mpt -n openwhisk
Error from server (BadRequest): container "invoker" in pod "invoker-b5mpt" is waiting to start: PodInitializing
Please suggest if I am doing something wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services