You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "tuanhoangth1603 (via GitHub)" <gi...@apache.org> on 2024/08/21 13:44:16 UTC

[I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

tuanhoangth1603 opened a new issue, #7517:
URL: https://github.com/apache/cloudstack/issues/7517

   Hello,
   I created a Kubernetes cluster on Cloudstack and used this guide (https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-provider.html) to deploy Traefik as an ingress controller. 
   1. According to the guide:
   _On successfully deploying the yaml file, a new Public IP Address in the same network as the cluster will be created. It will automatically have the firewall and port forwarding rules configured to distribute any traffic **amongst the cluster worker nodes**_
   
   However, upon inspecting the LB rules, I noticed that the **control node** was also added as a backend server ?
   ![image](https://github.com/apache/cloudstack/assets/38004836/c6668261-d0ab-44e8-bf30-aa077c306897)
   
   2. Furthermore, I created and deployed a file that defines a basic deployment, service, and ingress. However, when I curl http://k8s.lab.com.vn/, the response is "404 page not found". Is there a mistake in my YAML file? 
   Thanks for your help!
   
   ```
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: app
   spec:
     selector:
       app: app
     ports:
       - protocol: TCP
         port: 80
         targetPort: 80
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: app-deployment
     labels:
       app: app
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: app
     template:
       metadata:
         labels:
           app: app
       spec:
         containers:
         - name: app
           image: nginx:latest
           ports:
           - containerPort: 80
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: traefik-app-ingress
   spec:
     rules:
       - host: k8s.lab.com.vn
         http:
           paths:
             - path: /
               pathType: Prefix
               backend:
                 service:
                   name:  app
                   port:
                     number: 80
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2302403916

   and I also found this log in mgmt to be helpful
   ```2024-08-21 22:09:17,133 WARN  [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-47:ctx-4b9b2287 job-16540 ctx-d327ad2e) (logid:584f4248) API endpoint for Kubernetes cluster : tuanlab-k8s not available
   javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
   	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1701)
   	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1519)
   	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
   	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
   	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
   	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
   	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
   	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1613)
   	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1541)
   	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
   	at com.cloud.kubernetes.cluster.utils.KubernetesClusterUtil.isKubernetesClusterServerRunning(KubernetesClusterUtil.java:239)
   	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:546)
   	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1321)
   	at org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd.execute(CreateKubernetesClusterCmd.java:294)
   	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
   	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
   	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
   	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
   	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
   	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
   	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
   	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   Caused by: java.io.EOFException: SSL peer shut down incorrectly
   	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
   	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
   	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
   	... 28 more```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2302338447

   any logs on cloudstack management server and in  the pod `cloud-controller-manager` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [cloudstack] tuanhoangth1603 closed issue #7517: Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 closed issue #7517: Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0
URL: https://github.com/apache/cloudstack/issues/7517


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #7517: Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0
URL: https://github.com/apache/cloudstack/issues/7517


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2304762517

   Thanks for you idea.
   Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2304965546

   > Thanks for you idea.
   > Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.
   
   It is used to recreate ccm with latest 1.1.0
   If you have already installed 1.1.0, ignore it
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2304919818

   > Thanks for you idea.
   > Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.
   
   It is used to recreate ccm with latest 1.1.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [cloudstack] tuanhoangth1603 commented on issue #7517: Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-1553002447

   I follow this guide and the issue was solved.
   https://doc.traefik.io/traefik/getting-started/install-traefik/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2303599814

   Hi @weizhouapache,
   
   Do you have any suggestion for this issue ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2304023277

   > any logs on cloudstack management server and in the pod `cloud-controller-manager` ?
   
   @tuanhoangth1603 
   it looks like failed to get the cks nodes/vms `"tuanlab-k8s-control-1917579d77d"` and `"tuanlab-k8s-node-191757a899c"`
   
   can you try with cloudstack-kubernetes-provider v1.1.0 ?
   
   1. delete cloudstack-kubernetes-provider 1.0.0
   kubectl delete -f
   https://raw.githubusercontent.com/apache/cloudstack-kubernetes-provider/main/deployment.yaml
   
   2.  install 1.1.0
   kubectl apply -f
   https://github.com/apache/cloudstack-kubernetes-provider/releases/download/v1.1.0/deployment.yaml


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [cloudstack] weizhouapache commented on issue #7517: Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0

Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-1552890663

   @tuanhoangth1603 
   have you solved the issue ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2302104254

   Hello,
   
   Today I following this procedure to deploy k8s with traefik ingress. 
   I am facing an error with traefix ingress: "Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value" 
   Here is full describe of service:
   ```tuanlab@rke-nginx-proxy:~$ kubectl describe svc traefik
   Name:                     traefik
   Namespace:                default
   Labels:                   app.kubernetes.io/instance=traefik-default
                             app.kubernetes.io/managed-by=Helm
                             app.kubernetes.io/name=traefik
                             helm.sh/chart=traefik-30.1.0
   Annotations:              meta.helm.sh/release-name: traefik
                             meta.helm.sh/release-namespace: default
                             service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: enabled
   Selector:                 app.kubernetes.io/instance=traefik-default,app.kubernetes.io/name=traefik
   Type:                     LoadBalancer
   IP Family Policy:         SingleStack
   IP Families:              IPv4
   IP:                       10.107.132.3
   IPs:                      10.107.132.3
   Port:                     http  80/TCP
   TargetPort:               http/TCP
   NodePort:                 http  30893/TCP
   Endpoints:                <none>
   Port:                     https  443/TCP
   TargetPort:               https/TCP
   NodePort:                 https  32256/TCP
   Endpoints:                <none>
   Session Affinity:         None
   External Traffic Policy:  Cluster
   Events:
     Type     Reason                  Age                  From                Message
     ----     ------                  ----                 ----                -------
     Normal   EnsuringLoadBalancer    58s (x6 over 2m14s)  service-controller  Ensuring load balancer
     Warning  SyncLoadBalancerFailed  58s (x6 over 2m14s)  service-controller  Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0 [cloudstack]

Posted by "tuanhoangth1603 (via GitHub)" <gi...@apache.org>.
tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2302392513

   > any logs on cloudstack management server and in the pod `cloud-controller-manager` ?
   
   Thanks for your reply. Here is logs of CCM
   ```I0821 15:10:39.921244       1 serving.go:313] Generated self-signed cert in-memory
   W0821 15:11:05.221926       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
   I0821 15:11:05.420413       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
   I0821 15:11:05.422112       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
   I0821 15:11:05.422169       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
   I0821 15:11:05.422223       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
   I0821 15:11:05.422240       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
   I0821 15:11:05.618715       1 secure_serving.go:178] Serving securely on [::]:10258
   I0821 15:11:05.618813       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
   I0821 15:11:05.619351       1 tlsconfig.go:240] Starting DynamicServingCertificateController
   I0821 15:11:05.918307       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
   I0821 15:11:05.918307       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
   I0821 15:11:06.018794       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"79917607-9b73-462e-bf49-76a5cb88c24b", APIVersion:"v1", ResourceVersion:"785", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572 became leader
   E0821 15:11:06.018661       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"33584655-e620-4127-9ce5-e1ac08e95df8", ResourceVersion:"786", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63859849865, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc0001a6340), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a6380)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime
 :(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572 became leader'
   I0821 15:11:06.118337       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
   I0821 15:11:06.120413       1 node_controller.go:110] Sending events to api server.
   I0821 15:11:06.120516       1 controllermanager.go:247] Started "cloud-node"
   I0821 15:11:06.220667       1 node_lifecycle_controller.go:78] Sending events to api server
   I0821 15:11:06.220758       1 controllermanager.go:247] Started "cloud-node-lifecycle"
   I0821 15:11:06.319991       1 controllermanager.go:247] Started "service"
   I0821 15:11:06.320047       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
   W0821 15:11:06.320064       1 controllermanager.go:244] Skipping "route"
   I0821 15:11:06.418280       1 controller.go:208] Starting service controller
   I0821 15:11:06.418313       1 shared_informer.go:223] Waiting for caches to sync for service
   I0821 15:11:06.918866       1 shared_informer.go:230] Caches are synced for service
   E0821 15:11:08.719857       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:11:08.721984       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:11:08.818466       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:11:08.821324       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:11:08.823046       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:11:08.920077       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:16:09.221094       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:16:09.319975       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:16:09.321029       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:16:09.323385       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:16:09.324513       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:16:09.327037       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:21:09.922209       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:21:10.019894       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:21:10.021178       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:21:10.118357       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:21:10.120241       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:21:10.122821       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   I0821 15:25:38.304728       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3023", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:25:38.519534       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:25:38.519646       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3023", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:25:43.520304       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:25:43.521805       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:25:43.521878       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:25:53.522501       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:25:53.524087       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:25:53.524171       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   E0821 15:26:10.619873       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:26:10.622562       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:26:10.718288       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:26:10.721020       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:26:10.722174       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:26:10.724422       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   I0821 15:26:11.190439       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:26:11.191841       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:26:11.191884       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:26:13.524803       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:26:13.526348       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:26:13.526489       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:27:33.527261       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:27:33.721428       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:27:33.818172       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:30:13.723275       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
   E0821 15:30:13.921406       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   I0821 15:30:14.018213       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
   E0821 15:31:10.921484       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:31:11.119953       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:31:11.121277       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:31:11.123845       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   E0821 15:31:11.125066       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
   E0821 15:31:11.127270       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org