You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2020/12/30 08:52:16 UTC
[GitHub] [apisix-helm-chart] Caelebs opened a new issue #16: The Charts deployment apisix with etcd error
Caelebs opened a new issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16
When I run install directly, the container `wait-etcd` is always in the init state, and the log prompts bad address
```
waiting for etcd Wed Dec 30 08:33:51 UTC 2020
nc: bad address 'apisix-rax59c-etcd.helm-test.svc.cluster.local'
waiting for etcd Wed Dec 30 08:33:53 UTC 2020
nc: bad address 'apisix-rax59c-etcd.helm-test.svc.cluster.local'
waiting for etcd Wed Dec 30 08:33:55 UTC 2020
nc: bad address 'apisix-rax59c-etcd.helm-test.svc.cluster.local'
```
The configuration in the `values.yaml ` file is as follows, without any modification, I am not sure if I need to modify the `host`?
```
etcd:
enabled: true
host: etcd.host
prefix: "/apisix"
timeout: 30
```
The current situation is that when `enable ` is set to `false ` and an external etcd address is configured, apisix can be deployed and run normally. What do I need to modify so that apisix and etcd can be deployed at the same time? and kubectl describe pod output is as follows
```
Name: apisix-rax59c-bd7c7f8d6-2lclh
Namespace: helm-test
Priority: 0
Node: node04/10.10.16.242
Start Time: Wed, 30 Dec 2020 16:29:12 +0800
Labels: app.kubernetes.io/instance=apisix-rax59c
app.kubernetes.io/name=apisix
pod-template-hash=bd7c7f8d6
Annotations: kubernetes.io/limit-ranger:
LimitRanger plugin set: memory request for container apisix; memory limit for container apisix; cpu, memory request for init container wai...
Status: Pending
IP: 10.20.4.107
IPs:
IP: 10.20.4.107
Controlled By: ReplicaSet/apisix-rax59c-bd7c7f8d6
Init Containers:
wait-etcd:
Container ID: docker://da4dfc52a9dad093d903a7793a1bc532ff549d0bb4f644858a10c49d7c6728a4
Image: busybox:1.28
Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z apisix-rax59c-etcd.helm-test.svc.cluster.local 2379; do echo waiting for etcd `date`; sleep 2; done;
State: Running
Started: Wed, 30 Dec 2020 16:29:14 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mfwp4 (ro)
Containers:
apisix:
Container ID:
Image: caelebs/apisix:2.1-1218
Image ID:
Ports: 9080/TCP, 9443/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 500Mi
Requests:
cpu: 50m
memory: 10Mi
Readiness: tcp-socket :9080 delay=10s timeout=1s period=10s #success=1 #failure=6
Environment: <none>
Mounts:
/usr/local/apisix/conf/config.yaml from apisix-config (rw,path="config.yaml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mfwp4 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
apisix-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: apisix-rax59c
Optional: false
default-token-mfwp4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mfwp4
Optional: false
QoS Class: Burstable
Node-Selectors: node=node04
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned helm-test/apisix-rax59c-bd7c7f8d6-2lclh to node04
Normal Pulled 19m kubelet, node04 Container image "busybox:1.28" already present on machine
Normal Created 19m kubelet, node04 Created container wait-etcd
Normal Started 19m kubelet, node04 Started container wait-etcd
```
## k8s cluster information
```
[root@master01 mnt]# grep -A 1 'clusterDNS' /var/lib/kubelet/config.yaml
clusterDNS:
- 10.96.0.10
[root@master01 mnt]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 117d
[root@master01 mnt]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master01 Ready master 117d v1.16.9 10.10.16.250 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node01 Ready node 117d v1.16.9 10.10.16.251 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node02 Ready node 117d v1.16.9 10.10.16.252 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node03 Ready node 115d v1.16.9 10.10.16.241 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node04 Ready node 117d v1.16.9 10.10.16.242 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node05 Ready node 117d v1.16.9 10.10.16.243 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node06 Ready node 117d v1.16.9 10.10.16.244 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node07 Ready node 117d v1.16.9 10.10.16.245 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node08 Ready node 117d v1.16.9 10.10.16.246 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node09 Ready node 116d v1.16.9 10.10.16.247 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
node10 Ready node 116d v1.16.9 10.10.16.248 <none> CentOS Linux 7 (Core) 5.8.5-1.el7.elrepo.x86_64 docker://18.9.6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752829149
apisix pods are always in init state and it seems that no etcd pod is created,
```
[root@master01 mnt]# kubectl get pods apisix-rax59c-bd7c7f8d6-2lclh -n helm-test
NAME READY STATUS RESTARTS AGE
apisix-rax59c-bd7c7f8d6-2lclh 0/1 Init:0/1 0 18h
```
```
[root@master01 mnt]# kubectl get svc -n helm-test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
apisix-rax59c-gateway NodePort 10.106.74.60 <none> 80:31820/TCP 18h
```
```
[root@master01 mnt]# kubectl describe apisix-rax59c-bd7c7f8d6-2lclh -n helm-test
error: the server doesn't have a resource type "apisix-rax59c-bd7c7f8d6-2lclh"
[root@master01 mnt]# kubectl describe pod apisix-rax59c-bd7c7f8d6-2lclh -n helm-test
Name: apisix-rax59c-bd7c7f8d6-2lclh
Namespace: helm-test
Priority: 0
Node: node04/10.10.16.242
Start Time: Wed, 30 Dec 2020 16:29:12 +0800
Labels: app.kubernetes.io/instance=apisix-rax59c
app.kubernetes.io/name=apisix
pod-template-hash=bd7c7f8d6
Annotations: kubernetes.io/limit-ranger:
LimitRanger plugin set: memory request for container apisix; memory limit for container apisix; cpu, memory request for init container wai...
Status: Pending
IP: 10.20.4.107
IPs:
IP: 10.20.4.107
Controlled By: ReplicaSet/apisix-rax59c-bd7c7f8d6
Init Containers:
wait-etcd:
Container ID: docker://da4dfc52a9dad093d903a7793a1bc532ff549d0bb4f644858a10c49d7c6728a4
Image: busybox:1.28
Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z apisix-rax59c-etcd.helm-test.svc.cluster.local 2379; do echo waiting for etcd `date`; sleep 2; done;
State: Running
Started: Wed, 30 Dec 2020 16:29:14 +0800
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mfwp4 (ro)
Containers:
apisix:
Container ID:
Image: caelebs/apisix:2.1-1218
Image ID:
Ports: 9080/TCP, 9443/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 500Mi
Requests:
cpu: 50m
memory: 10Mi
Readiness: tcp-socket :9080 delay=10s timeout=1s period=10s #success=1 #failure=6
Environment: <none>
Mounts:
/usr/local/apisix/conf/config.yaml from apisix-config (rw,path="config.yaml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mfwp4 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
apisix-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: apisix-rax59c
Optional: false
default-token-mfwp4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mfwp4
Optional: false
QoS Class: Burstable
Node-Selectors: node=node04
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
```
There is an etcd container running on node node4, but `bad address` is always output in the log
```
[root@node04 ~]# docker ps | grep wait-etcd
da4dfc52a9da 8c811b4aec35 "sh -c 'until nc -z …" 19 hours ago Up 19 hours k8s_wait-etcd_apisix-rax59c-bd7c7f8d6-2lclh_helm-test_ae45fd11-d508-4f6f-8dcd-6e93f45a9af4_0
```
```
nc: bad address 'apisix-rax59c-etcd.helm-test.svc.cluster.local'
waiting for etcd Wed Dec 30 20:53:07 UTC 2020
nc: bad address 'apisix-rax59c-etcd.helm-test.svc.cluster.local'
waiting for etcd Wed Dec 30 20:53:09 UTC 2020
```
Are there any other logs I need to provide?
@tokers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752833072
Do you mean I need to deploy etcd service separately? I would have thought that directly helm install apisix can automatically deploy apisix and dependent components etcd, Because I saw the `dependencies` node in `Chart.yaml`, I thought that etcd would be deployed automatically.
```
dependencies:
- name: etcd
version: 5.2.1
repository: https://charts.bitnami.com/bitnami
condition: etcd.enabled
```
Thanks Reply @tokers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tokers commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752811241
@Caelebs What about the running situation of etcd pods? Could you also paste the pod details and service details of etcd?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-755034557
> Sorry for late reply. @Caelebs
> Follow the [doc](https://github.com/apache/apisix-helm-chart#dependencies), we need to update the dependency before helm install.
Thanks for the reply. Later, I added the dependency etcd to the charts directory and the deployment was successful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] gxthrj commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-754729521
Sorry for late reply.
Follow the [doc](https://github.com/apache/apisix-helm-chart#dependencies), we need to update the dependency before helm install.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-755034237
> 抱歉回复晚了。@Caelebs
> 按照[doc进行操作](https://github.com/apache/apisix-helm-chart#dependencies)，我们需要在安装头盔之前更新依赖项。
Thanks for the reply. Later, I added the dependency etcd to the charts directory and the deployment was successful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tokers commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752830949
The etcd service is not deployed? Then of course the `nc` program in apisix initContainer to probe etcd will fail.
> There is an etcd container running on node node4, but bad address is always output in the log
This is the init container in apisix pod, not the etcd container.
It seems the etcd is not deployed correctly, be sure:
1) follow the correct step to install apisix by helm chart
2) check the status of etcd deployment, service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] membphis commented on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752651733
@gxthrj please take a look at this issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] tokers edited a comment on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
tokers edited a comment on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-752830949
The etcd service is not deployed? Then of course the `nc` program in apisix initContainer to probe etcd will fail.
> There is an etcd container running on node node4, but bad address is always output in the log
This is the init container in apisix pod, not the etcd container.
It seems the etcd is not deployed correctly:
1) follow the correct step to install apisix by helm chart
2) check the status of etcd deployment, service
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs removed a comment on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs removed a comment on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-755034237
> 抱歉回复晚了。@Caelebs
> 按照[doc进行操作](https://github.com/apache/apisix-helm-chart#dependencies)，我们需要在安装头盔之前更新依赖项。
Thanks for the reply. Later, I added the dependency etcd to the charts directory and the deployment was successful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] gxthrj edited a comment on issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
gxthrj edited a comment on issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16#issuecomment-754729521
Sorry for late reply. @Caelebs
Follow the [doc](https://github.com/apache/apisix-helm-chart#dependencies), we need to update the dependency before helm install.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-helm-chart] Caelebs closed issue #16: The Charts deployment apisix with etcd error
Posted by GitBox <gi...@apache.org>.
Caelebs closed issue #16:
URL: https://github.com/apache/apisix-helm-chart/issues/16
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org