You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Eric Green <er...@gmail.com> on 2024/12/12 03:14:54 UTC
Kubernetes nginx ingress svc stuck in 'Pending'
I am currently successfully using the nginx ingress on AKS. But when I try to deploy it into Cloudstack Kubernetes, the EXTERNAL-PI is stuck in <pending> state. I've tried multiple ways to deploy it and am putting the correct annotations as documented in the Cloudstack Kubernetes provider but I'm still stumped — no IP address showing up. I must have something misconfigured but what it is, eludes me. Any suggestions?
Kubernetes versions:
:~$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4
How I deploy the nginx ingress:
helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
The config file nginx-cloudstack.yaml:
-------------
# For deploying nginx load balancer on Cloudstack
# Use command:
# helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
controller:
kind: DaemonSet
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: enabled
ingressClassResource:
default: true
extraArgs:
default-ssl-certificate: "default/mycert"
config:
use-proxy-protocol: "true"
-------------
The results:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.105.120.100 <pending> 80:32164/TCP,443:32429/TCP 16m
ingress-nginx-controller-admission ClusterIP 10.111.151.26 <none> 443/TCP 16m
Kubectl get events shows no issues:
19m Normal Scheduled pod/ingress-nginx-admission-create-bfdkq Successfully assigned default/ingress-nginx-admission-create-bfdkq to default-node-193b75945ef
19m Normal Pulling pod/ingress-nginx-admission-create-bfdkq Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f"
19m Normal Pulled pod/ingress-nginx-admission-create-bfdkq Successfully pulled image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" in 2.55s (2.55s including waiting)
19m Normal Created pod/ingress-nginx-admission-create-bfdkq Created container create
19m Normal Started pod/ingress-nginx-admission-create-bfdkq Started container create
19m Normal SuccessfulCreate job/ingress-nginx-admission-create Created pod: ingress-nginx-admission-create-bfdkq
18m Normal Completed job/ingress-nginx-admission-create Job completed
18m Normal Scheduled pod/ingress-nginx-admission-patch-x5wxz Successfully assigned default/ingress-nginx-admission-patch-x5wxz to default-node-193b75945ef
18m Normal Pulled pod/ingress-nginx-admission-patch-x5wxz Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" already present on machine
18m Normal Created pod/ingress-nginx-admission-patch-x5wxz Created container patch
18m Normal Started pod/ingress-nginx-admission-patch-x5wxz Started container patch
18m Normal SuccessfulCreate job/ingress-nginx-admission-patch Created pod: ingress-nginx-admission-patch-x5wxz
18m Normal Completed job/ingress-nginx-admission-patch Job completed
18m Normal Scheduled pod/ingress-nginx-controller-lkzt9 Successfully assigned default/ingress-nginx-controller-lkzt9 to default-node-193b759116c
18m Normal Pulling pod/ingress-nginx-controller-lkzt9 Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7"
18m Normal Pulled pod/ingress-nginx-controller-lkzt9 Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.128s (6.128s including waiting)
18m Normal Created pod/ingress-nginx-controller-lkzt9 Created container controller
18m Normal Started pod/ingress-nginx-controller-lkzt9 Started container controller
18m Normal RELOAD pod/ingress-nginx-controller-lkzt9 NGINX reload triggered due to a change in configuration
18m Normal Scheduled pod/ingress-nginx-controller-s6zqd Successfully assigned default/ingress-nginx-controller-s6zqd to default-node-193b75945ef
18m Normal Pulling pod/ingress-nginx-controller-s6zqd Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7"
18m Normal Pulled pod/ingress-nginx-controller-s6zqd Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.171s (6.171s including waiting)
18m Normal Created pod/ingress-nginx-controller-s6zqd Created container controller
18m Normal Started pod/ingress-nginx-controller-s6zqd Started container controller
18m Normal RELOAD pod/ingress-nginx-controller-s6zqd NGINX reload triggered due to a change in configuration
18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-s6zqd
18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-lkzt9
18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller
18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller
$
Similarly, kubectl logs on the two pods created on the two nodes shows no issues.
From output of 'helm template' the three things needed to hook into Cloudstack seem to be there, the ConfigMap saying use-proxy-protocol: "true", the Service saying
" service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled" ", and the inclusion of the ConfigMap into the DaemonSet:
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
data:
allow-snippet-annotations: "false"
use-proxy-protocol: "true"
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled"
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
spec:
type: LoadBalancer
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv4
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
containers:
- name: controller
image: registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --enable-metrics=false
- --default-ssl-certificate=default/mycert
securityContext:
...
---
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Eric Green <er...@gmail.com>.
Hi, to clarify, I no longer have the issue with the LoadBalancer stuck in ‘Pending’. The redeployment of my Kubernetes cluster while experimenting with the various parameters for deploying a Kubernetes cluster finally got me to a point where my LoadBalancer properly got an IP address.
I now have the nginx Ingress controller operating and have deployed an Ingress that is accepting traffic. I am also now getting traffic at my pods.
The simplest service / Deployment / pod that I have is a simple haproxy that is on a /icons/(.*) url that simply forwards to a S3 bucket. I have a README in that s3 bucket and I can do e.g.
curl https://myapp.mydomain.com/icons/README
and see the contents of the README. So everything is working now. I can put that command in a shell script loop and it returns my README every time so there’s not any performance issues now that the networking is finally networking.
All my other services / Deployments / pods also have health check URLs that similarly are now working. As you would expect, since I’ve had this working in Azure for months now with the same Helm chart other than that it’s now pulling images from my local private registry rather than from my Azure private registry.
The only issue I now have is that it took a significant amount of time – like, almost an hour – before Kubernetes started routing traffic to my pods. I will see whether that remains a problem when I get the CICD pipeline working. After a git checkin and image build it’s going to do a Helm update of the app to update the Deployment to pull in the updated image which in turn will trigger a rolling update, I will see whether that will disrupt the networking and whether I will have an issue with the app going offline after an update.
So I think I’m okay for the moment. If the networking delay proves a problem going forward, I will update this thread with whatever information I can find that might be applicable.
Kubernetes on Azure – 1.29.7
Kubernetes on Cloudstack – 1.28.4
Cloudstack version: 4.19.1.3
From: Kiran Chavala <ki...@shapeblue.com>
Date: Tuesday, December 17, 2024 at 9:15 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi Eric
Thanks for the information.
Let me try to reproduce the issue in my lab.
What is the cloudstack version and Kubernetes version you are using ?
Also could you please share the nginx ingress controller yaml
It’s documented in the example to use the cloud user
https://docs.cloudstack.apache.org/en/4.20.0.0/plugins/cloudstack-kubernetes-service.html#kubernetes-clusters
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 10:22 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi, I am using KVM with VLAN-based advanced networking. I have allocated VLANs 1000-2000 to VPC’s and looking at the virtual router, it’s using VLAN 1016. I successfully can ssh through the virtual router IP address to the nodes via the virtual router port redirects that were set up by Cloudstack when I instantiated the Kubernetes cluster (2222 and up, each node incrementing the port number). I did have to experiment a bit to find that the user name ‘cloud’ was what I needed to use to get into the nodes, that does not appear to be documented anywhere.
From: Kiran Chavala <ki...@shapeblue.com>
Date: Tuesday, December 17, 2024 at 8:26 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi Eric
Could you please let me know the hypervisor you are using with Cloudstack.
Is it KVM or Vmware ?
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 4:40 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Ugh. Literally the simplest possible configuration and it's stuck in 'Pending'. So it has to be an issue on the Cloudstack side? Let me get the configuration of that Kubernetes cluster....
Okay. I destroyed the Kubernetes cluster and re-created it trying various options until your two commands created a working LoadBalancer with an assigned IP address. Then I created a Nagios ingress controller using the described YAML and successfully created the nginx ingress controller with an IP address for its LoadBalancer service. Now to get a working Ingress that routes traffic through the Services to my pods... Wish me luck. (To reiterate, I already have all of this working on AKS, Azure Kubernetes Service, it's just Cloudstack's Kubernetes that's being a pain to me).
________________________________
From: Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>
Sent: Wednesday, December 11, 2024 7:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Kiran Chavala <ki...@shapeblue.com>.
Hi Eric
Thanks for the information.
Let me try to reproduce the issue in my lab.
What is the cloudstack version and Kubernetes version you are using ?
Also could you please share the nginx ingress controller yaml
It’s documented in the example to use the cloud user
https://docs.cloudstack.apache.org/en/4.20.0.0/plugins/cloudstack-kubernetes-service.html#kubernetes-clusters
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 10:22 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi, I am using KVM with VLAN-based advanced networking. I have allocated VLANs 1000-2000 to VPC’s and looking at the virtual router, it’s using VLAN 1016. I successfully can ssh through the virtual router IP address to the nodes via the virtual router port redirects that were set up by Cloudstack when I instantiated the Kubernetes cluster (2222 and up, each node incrementing the port number). I did have to experiment a bit to find that the user name ‘cloud’ was what I needed to use to get into the nodes, that does not appear to be documented anywhere.
From: Kiran Chavala <ki...@shapeblue.com>
Date: Tuesday, December 17, 2024 at 8:26 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi Eric
Could you please let me know the hypervisor you are using with Cloudstack.
Is it KVM or Vmware ?
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 4:40 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Ugh. Literally the simplest possible configuration and it's stuck in 'Pending'. So it has to be an issue on the Cloudstack side? Let me get the configuration of that Kubernetes cluster....
Okay. I destroyed the Kubernetes cluster and re-created it trying various options until your two commands created a working LoadBalancer with an assigned IP address. Then I created a Nagios ingress controller using the described YAML and successfully created the nginx ingress controller with an IP address for its LoadBalancer service. Now to get a working Ingress that routes traffic through the Services to my pods... Wish me luck. (To reiterate, I already have all of this working on AKS, Azure Kubernetes Service, it's just Cloudstack's Kubernetes that's being a pain to me).
________________________________
From: Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>
Sent: Wednesday, December 11, 2024 7:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Eric Green <er...@gmail.com>.
Hi, I am using KVM with VLAN-based advanced networking. I have allocated VLANs 1000-2000 to VPC’s and looking at the virtual router, it’s using VLAN 1016. I successfully can ssh through the virtual router IP address to the nodes via the virtual router port redirects that were set up by Cloudstack when I instantiated the Kubernetes cluster (2222 and up, each node incrementing the port number). I did have to experiment a bit to find that the user name ‘cloud’ was what I needed to use to get into the nodes, that does not appear to be documented anywhere.
From: Kiran Chavala <ki...@shapeblue.com>
Date: Tuesday, December 17, 2024 at 8:26 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Hi Eric
Could you please let me know the hypervisor you are using with Cloudstack.
Is it KVM or Vmware ?
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 4:40 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Ugh. Literally the simplest possible configuration and it's stuck in 'Pending'. So it has to be an issue on the Cloudstack side? Let me get the configuration of that Kubernetes cluster....
Okay. I destroyed the Kubernetes cluster and re-created it trying various options until your two commands created a working LoadBalancer with an assigned IP address. Then I created a Nagios ingress controller using the described YAML and successfully created the nginx ingress controller with an IP address for its LoadBalancer service. Now to get a working Ingress that routes traffic through the Services to my pods... Wish me luck. (To reiterate, I already have all of this working on AKS, Azure Kubernetes Service, it's just Cloudstack's Kubernetes that's being a pain to me).
________________________________
From: Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>
Sent: Wednesday, December 11, 2024 7:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Kiran Chavala <ki...@shapeblue.com>.
Hi Eric
Could you please let me know the hypervisor you are using with Cloudstack.
Is it KVM or Vmware ?
Regards
Kiran
From: Eric Green <er...@gmail.com>
Date: Wednesday, 18 December 2024 at 4:40 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Ugh. Literally the simplest possible configuration and it's stuck in 'Pending'. So it has to be an issue on the Cloudstack side? Let me get the configuration of that Kubernetes cluster....
Okay. I destroyed the Kubernetes cluster and re-created it trying various options until your two commands created a working LoadBalancer with an assigned IP address. Then I created a Nagios ingress controller using the described YAML and successfully created the nginx ingress controller with an IP address for its LoadBalancer service. Now to get a working Ingress that routes traffic through the Services to my pods... Wish me luck. (To reiterate, I already have all of this working on AKS, Azure Kubernetes Service, it's just Cloudstack's Kubernetes that's being a pain to me).
________________________________
From: Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>
Sent: Wednesday, December 11, 2024 7:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Eric Green <er...@gmail.com>.
Ugh. Literally the simplest possible configuration and it's stuck in 'Pending'. So it has to be an issue on the Cloudstack side? Let me get the configuration of that Kubernetes cluster....
Okay. I destroyed the Kubernetes cluster and re-created it trying various options until your two commands created a working LoadBalancer with an assigned IP address. Then I created a Nagios ingress controller using the described YAML and successfully created the nginx ingress controller with an IP address for its LoadBalancer service. Now to get a working Ingress that routes traffic through the Services to my pods... Wish me luck. (To reiterate, I already have all of this working on AKS, Azure Kubernetes Service, it's just Cloudstack's Kubernetes that's being a pain to me).
________________________________
From: Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>
Sent: Wednesday, December 11, 2024 7:22 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Kubernetes nginx ingress svc stuck in 'Pending'
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
Re: Kubernetes nginx ingress svc stuck in 'Pending'
Posted by Jayanth Babu Reddy <ja...@nxtgen.com.INVALID>.
Just to confirm if there's not any misconfiguration at ACS, try doing these and let us know if you get the IP assigned.
$ kubectl run --image=nginx nginx
$ kubectl expose pod/nginx --type=LoadBalancer --port=80
For your nginx ingress controller service of type Load Balancer, what does $ kubectl describe svc ingress-nginx-controller say?
Thanks,
Jayanth
________________________________
From: Eric Green <er...@gmail.com>
Sent: Thursday, December 12, 2024 8:46:12 am
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Kubernetes nginx ingress svc stuck in 'Pending'
I am currently successfully using the nginx ingress on AKS. But when I try to deploy it into Cloudstack Kubernetes, the EXTERNAL-PI is stuck in <pending> state. I've tried multiple ways to deploy it and am putting the correct annotations as documented in the Cloudstack Kubernetes provider but I'm still stumped — no IP address showing up. I must have something misconfigured but what it is, eludes me. Any suggestions?
Kubernetes versions:
:~$ kubectl version
Client Version: v1.29.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4
How I deploy the nginx ingress:
helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
The config file nginx-cloudstack.yaml:
-------------
# For deploying nginx load balancer on Cloudstack
# Use command:
# helm install -f nginx-cloudstack.yaml ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx
controller:
kind: DaemonSet
service:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: enabled
ingressClassResource:
default: true
extraArgs:
default-ssl-certificate: "default/mycert"
config:
use-proxy-protocol: "true"
-------------
The results:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.105.120.100 <pending> 80:32164/TCP,443:32429/TCP 16m
ingress-nginx-controller-admission ClusterIP 10.111.151.26 <none> 443/TCP 16m
Kubectl get events shows no issues:
19m Normal Scheduled pod/ingress-nginx-admission-create-bfdkq Successfully assigned default/ingress-nginx-admission-create-bfdkq to default-node-193b75945ef
19m Normal Pulling pod/ingress-nginx-admission-create-bfdkq Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f"
19m Normal Pulled pod/ingress-nginx-admission-create-bfdkq Successfully pulled image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" in 2.55s (2.55s including waiting)
19m Normal Created pod/ingress-nginx-admission-create-bfdkq Created container create
19m Normal Started pod/ingress-nginx-admission-create-bfdkq Started container create
19m Normal SuccessfulCreate job/ingress-nginx-admission-create Created pod: ingress-nginx-admission-create-bfdkq
18m Normal Completed job/ingress-nginx-admission-create Job completed
18m Normal Scheduled pod/ingress-nginx-admission-patch-x5wxz Successfully assigned default/ingress-nginx-admission-patch-x5wxz to default-node-193b75945ef
18m Normal Pulled pod/ingress-nginx-admission-patch-x5wxz Container image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f" already present on machine
18m Normal Created pod/ingress-nginx-admission-patch-x5wxz Created container patch
18m Normal Started pod/ingress-nginx-admission-patch-x5wxz Started container patch
18m Normal SuccessfulCreate job/ingress-nginx-admission-patch Created pod: ingress-nginx-admission-patch-x5wxz
18m Normal Completed job/ingress-nginx-admission-patch Job completed
18m Normal Scheduled pod/ingress-nginx-controller-lkzt9 Successfully assigned default/ingress-nginx-controller-lkzt9 to default-node-193b759116c
18m Normal Pulling pod/ingress-nginx-controller-lkzt9 Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7"
18m Normal Pulled pod/ingress-nginx-controller-lkzt9 Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.128s (6.128s including waiting)
18m Normal Created pod/ingress-nginx-controller-lkzt9 Created container controller
18m Normal Started pod/ingress-nginx-controller-lkzt9 Started container controller
18m Normal RELOAD pod/ingress-nginx-controller-lkzt9 NGINX reload triggered due to a change in configuration
18m Normal Scheduled pod/ingress-nginx-controller-s6zqd Successfully assigned default/ingress-nginx-controller-s6zqd to default-node-193b75945ef
18m Normal Pulling pod/ingress-nginx-controller-s6zqd Pulling image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7"
18m Normal Pulled pod/ingress-nginx-controller-s6zqd Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7" in 6.171s (6.171s including waiting)
18m Normal Created pod/ingress-nginx-controller-s6zqd Created container controller
18m Normal Started pod/ingress-nginx-controller-s6zqd Started container controller
18m Normal RELOAD pod/ingress-nginx-controller-s6zqd NGINX reload triggered due to a change in configuration
18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-s6zqd
18m Normal SuccessfulCreate daemonset/ingress-nginx-controller Created pod: ingress-nginx-controller-lkzt9
18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller
18m Normal CREATE configmap/ingress-nginx-controller ConfigMap default/ingress-nginx-controller
$
Similarly, kubectl logs on the two pods created on the two nodes shows no issues.
From output of 'helm template' the three things needed to hook into Cloudstack seem to be there, the ConfigMap saying use-proxy-protocol: "true", the Service saying
" service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled" ", and the inclusion of the ConfigMap into the DaemonSet:
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
data:
allow-snippet-annotations: "false"
use-proxy-protocol: "true"
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: "enabled"
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
spec:
type: LoadBalancer
ipFamilyPolicy: SingleStack
ipFamilies:
- IPv4
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: default
spec:
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
revisionHistoryLimit: 10
minReadySeconds: 0
template:
metadata:
labels:
helm.sh/chart: ingress-nginx-4.11.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: "1.11.3"
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
spec:
dnsPolicy: ClusterFirst
containers:
- name: controller
image: registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --enable-metrics=false
- --default-ssl-certificate=default/mycert
securityContext:
...
---
Disclaimer *** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not authorised to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful and strictly prohibited. This e-mail may contain viruses. NxtGen Datacenter & Cloud Technologies Private Ltd (“NxtGen”) has taken every reasonable precaution to minimize this risk but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. NxtGen reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the NxtGen e-mail system. *** End of Disclaimer ***NXTGEN***