You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Lakshmi2k1 (via GitHub)" <gi...@apache.org> on 2024/06/06 05:56:44 UTC
[I] help request: Apisix ETCD going into Crash loop back off [apisix]
Lakshmi2k1 opened a new issue, #11338:
URL: https://github.com/apache/apisix/issues/11338
### Description
Hello,
I have deployed apisix 2.7.0 Helm chart and out of three etcd pods, two are going into crash loop back off error which affects the ingress created for other deployments.
<img width="674" alt="image" src="https://github.com/apache/apisix/assets/59364250/d4c834ba-146c-4bb6-b369-76f7b861ef42">
The logs show the following details,
Master (etcd pod in running state)
"msg":"rejected stream from remote peer because it was removed","local-member-id"
Other pods (etcd pods in crash loop back off state)
"failed to publish local member to cluster through raft","local-member-id":"2c16fb63879f0d98","local-member-attributes":"{Name:apisix-etcd-1 ClientURLs:[http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2379/ http://apisix-etcd.apisix.svc.cluster.local:2379]}","request-path":"/0/members/2c16fb63879f0d98/attributes","publish-timeout":"7s","error":"etcdserver: request cancelled"
Currently stuck in this, let me know if anyone has faced this and has any fix for this
### Environment
- APISIX version (run `apisix version`): 2.7.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Thilip707 (via GitHub)" <gi...@apache.org>.
Thilip707 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2160191455
Thanks its working now! I have another doubt did u configure kubernetes service discovery in apisix?
i configured this but i got this error handle_upstream(): failed to set upstream: discovery kubernetes is uninitialized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Thilip707 (via GitHub)" <gi...@apache.org>.
Thilip707 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2255071070
just increase client size and check it will work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2153976854
> Do you have a strong requirement to use `2.7.0`? I'm using the latest and pods are starting normally.
The most recent version is 2.8.0 (released in Jun 04, 2024). So, was using one version prior to that which got released on April. May I know which version of helm chart you're using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "minedetector (via GitHub)" <gi...@apache.org>.
minedetector commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2295937821
We had a very similar issue.
All of our etcd pods were going into crashloopbackoff and had only these warnings.
```
Cluster not healthy, not adding self to cluster for now, keeping trying...
```
We created apisix and etcd through the helm chart and for us the issue was that even though we re-created the StatefulSet and deleted the PVC-s for a fresh start the `ETCD_INITIAL_CLUSTER_STATE` ENV was still set to `existing`.
Changed it to `new` scheduled STS to 0 and then 3 again and it started working for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2277150654
> > @sudhir649 @BadTorro Deploying new version of etcd worked for me initially but whenever the node is scaled down and scaled up, one of the etcd pods is going into crashloop. But since other two etcd pods were running, it wasn't affecting the route and upstream creations. Yet, we are about to use it in production environment, so wish there is a permanent fix. After reading the solution @sudhir649 you have pointed out, I have few questions. 1. Won't deleting the PVC cause loss of data that apisix needs. 2. For second solution, I feel it's good to give a try. (n-1)/2 , in my case number of etcd replicas is 3, so according to this even if one pod is crashing, the disaster recovery cron will run and take backup of pvc. But as you mentioned when two pods were crashing there was no change, when third pod also crashed then all three pods came to running state. But in my case only one or rarely two pods crashing. If you have any inputs let me know. Thanks in advance!
>
> Regarding to that, I managed to get it work by basically:
>
> * Deploying [longhorn](https://longhorn.io/) storage solution to the cluster
> * Configured rancher desktop based on[ this guide](https://medium.com/@thizaom/rancher-desktop-with-rancher-longhorn-280e687f5022) to have open-iscsi in place and useable
> * changed the storageclass in the dedicated etcd sub-chart and related values.yaml file to "longhorn"
>
> ```
> persistence:
> enabled: true
> storageClass: "longhorn"
> ```
>
> * started everything with "tilt up"
>
> Currently keeps on running and did not crash since. However, we are now as well checking if the [Bitnami chart](https://artifacthub.io/packages/helm/bitnami/apisix) runs out of the box...
Hi @BadTorro , How was the experince after deploying the disater recovery? For us its working fine so we replicate it in all the envs.
Regards,
Sudhir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Thilip707 (via GitHub)" <gi...@apache.org>.
Thilip707 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2255049121
> @sudhir649, We are facing one more error in apisix. We use openid-connect plugin for authentication and authorization in the ApisixPluginConfig. When we try to hit the ingress of application, it gives 431 (Header too large) error. We tried removing few headers but it was breaking UI of application, so is there a way to solve this? Have you come across similar issue before?
did u use nginx? if u use nginx add this in nginx
client_max_body_size 2G;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Joeydelarago (via GitHub)" <gi...@apache.org>.
Joeydelarago commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2552935507
This issue started occurring again for me, so I ended up installing etcd separately.
I installed etcd. I only updated the etcd values.yaml to increase the replication factor from 1 -> 3
```
helm install etcd-apisix bitnami/etcd \
--namespace <NAMESPACE> \
--values etcd-values.yaml
```
Then I updated the apisix values.yaml and did a helm upgrade apisix.
```
externalEtcd:
host:
- http://etcd-apisix.<NAMESPACE>.svc.cluster.local:2379
user: root
existingSecret: "etcd-apisix"
secretPasswordKey: "etcd-root-password"
...
etcd:
enabled: false
```
My solution for the issue is still the same as I mentioned above. Helm uninstall, delete etcd PVC, helm reinstall. However, with etcd separated, this can be done without taking down apisix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "flea1lt (via GitHub)" <gi...@apache.org>.
flea1lt commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2151710713
I think you need to try `etcdctl member list` first. This will help you verify if the member ID of the crashing pods matches the IDs from the etcdctl.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2268236435
> @Lakshmi2k1
>
> 1. for deleting the pvc it's depends what data are you storing into it. In my case Or generally we are storing the routes only so if I deleted it will restore again once new pvc is created.
> 2. In the documentaion they have mentioned more than (n-1) /2 .It means when more than 1 pod (atleast 2 if you have 3 etcd pods) will fail then automatically pods will try to recover. Recently in our QA env all the pods were down so it's better to implement disater recovery.
I have enabled disaster recovery and deployed the helm chart, but this time not just etcd was crashing, the apisix pod stuck in init container, apisix ingress controller was crashing and the snapshot pod was also in error state. So, I rolled back to previous revision again after observing the pod status doesn't seem to change for a long time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "BadTorro (via GitHub)" <gi...@apache.org>.
BadTorro commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2327295461
> Hi @BadTorro , How was the experince after deploying the disater recovery? For us its working fine so we replicate it in all the envs.
>
> Regards, Sudhir
apologizes for the late reply @sudhir649 , but we ended up using the bitnami chart and customized it to our needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2228903335
Hi @Lakshmi2k1 Still are you facing the same issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Joeydelarago (via GitHub)" <gi...@apache.org>.
Joeydelarago commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2462411316
> Hi @Joeydelarago , when you delete the etcd, what data is gone ya? all route is gone or is still exist?
I actually encountered this when setting up a new environment, so it was not a concern for me. Also I have done my configuration via yaml files instead of the API.
If you need the data, you can always mount the pvcs to a dummy deployment and copy the files to local with kubectl cp. Then delete the pvc and apply apisix. Then use kubectl cp to copy the important files to the newly created pvc. I can't guarantee it will work though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "pietrogoddibit2win (via GitHub)" <gi...@apache.org>.
pietrogoddibit2win commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2326259199
The problem is the same here https://github.com/bitnami/charts/issues/16069
When a Pod present in ETCD_INITIAL_CLUSTER is schduled in a new node it starts with and empty PVC, so the pod is not able to join the cluster anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2254888601
@sudhir649
Thanks Sudhir, I'll try the same from my end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2253859047
@sudhir649 @BadTorro
Deploying new version of etcd worked for me initially but whenever the node is scaled down and scaled up, one of the etcd pods is going into crashloop. But since other two etcd pods were running, it wasn't affecting the route and upstream creations. Yet, we are about to use it in production environment, so wish there is a permanent fix. After reading the solution @sudhir649 you have pointed out, I have few questions.
1. Won't deleting the PVC cause loss of data that apisix needs.
2. For second solution, I feel it's good to give a try. (n-1)/2 , in my case number of etcd replicas is 3, so according to this even if one pod is crashing, the disaster recovery cron will run and take backup of pvc. But as you mentioned when two pods were crashing there was no change, when third pod also crashed then all three pods came to running state. But in my case only one or rarely two pods crashing. If you have any inputs let me know. Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "BadTorro (via GitHub)" <gi...@apache.org>.
BadTorro commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2251272663
How do you mean? Do you have some more specifics to that?
We're currently using it on a local development environment and etcd boots
with 3 nodes, but 2 always keep failing.
From time to time I need to shutdown the entire environment and restart it
again to get it working again.
Using it within https://tilt.dev/
thanks
Am Do., 25. Juli 2024 um 21:25 Uhr schrieb sudhir649 <
***@***.***>:
> Hi @BadTorro <https://github.com/BadTorro> try to enable the disaster
> recovery cron job.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/apisix/issues/11338#issuecomment-2251245375>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A4UABJKQNHR74EPNW5BG55LZOFGKHAVCNFSM6AAAAABI4ARYQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRGI2DKMZXGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "ch-gr (via GitHub)" <gi...@apache.org>.
ch-gr commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3192142893
> 4. If you are deploying APISIX on Kubernetes, we recommend controlling the `apache/apisix` gateway via the **APISIX Ingress Controller 2.0+** The new design does not use ETCD, so you will not encounter these issues again. Please read https://github.com/apache/apisix-ingress-controller/blob/master/docs/en/latest/upgrade-guide.md and https://docs.api7.ai/apisix/reference/apisix-ingress-controller/examples to understand the differences and upgrade steps.
@juzhiyuan for Kubernetes, "[standalone mode without etcd](https://github.com/apache/apisix-ingress-controller/blob/2.0.0-rc3/docs/en/latest/install.md#install-apisix-and-apisix-ingress-controller-standalone-api-driven-mode)" seems to work:
```bash
helm install apisix \
--namespace ingress-apisix \
--create-namespace \
--set apisix.deployment.role=traditional \
--set apisix.deployment.role_traditional.config_provider=yaml \
--set etcd.enabled=false \
--set ingress-controller.enabled=true \
--set ingress-controller.config.provider.type=apisix-standalone \
--set ingress-controller.apisix.adminService.namespace=ingress-apisix \
--set ingress-controller.gatewayProxy.createDefault=true \
apisix/apisix
```
But the documentation has an experimental tag in the [docs](https://github.com/apache/apisix/blob/3.13.0/docs/en/latest/deployment-modes.md#api-driven-experimental):
```text
This mode is experimental, please do not rely on it in your production environment. We use it to validate certain specific workloads and if it is appropriate we will turn it into an officially supported feature, otherwise it will be removed.
```
And the current apisix helm chart (v2.11.5) uses a release candidate for the apisix-ingress-controller helm chart (v2.0.0-rc3).
Is there a timeline for a GA release, with "standalone mode without etcd" as an officially supported feature, suitable for production environments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3163224916
As this page is too long, GitHub hides some information by default.
Please read https://github.com/apache/apisix/issues/11338#issuecomment-3162166205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2154640979
> There is one running etcd pod, run `etcdctl member list` after exec into the pod. And check the logs of crashed etcd pods, normally there was member id it used.
>
> BTW, I think it's more likely a etcd problem.
Hello,
I have tried it, this is what i got
I have no name!@apisix-etcd-2:/opt/bitnami/etcd$ etcdctl member list
3ff1b5cd453a87df, started, apisix-etcd-2, http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380, http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2379,http://apisix-etcd.apisix.svc.cluster.local:2379, false and
this is the member id I found in the logs of crashing pods [local-member-id":"2c16fb63879f0d98"]. I also tried disabling apisix etcd and used an external etcd but it was not able to integrate with the running etcd pod. I'm trying to fix that. Share if anything you know could help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "pietrogoddibit2win (via GitHub)" <gi...@apache.org>.
pietrogoddibit2win commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2326297168
> JFYI : `removeMemberOnContainerTermination:false` didn't help me on AKS cluster v.1.28.9 (docker.io/bitnami/etcd:3.5.10-debian-11-r2) `{"level":"warn","ts":"2024-09-03T11:26:04.835351Z","caller":"etcdserver/server.go:1127","msg":"server error","error":"the member has been permanently removed from the cluster"}` Any ideas? :-)
neither in GKE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2253991941
@Lakshmi2k1
1) for deleting the pvc it's depends what data are you storing into it. In my case Or generally we are storing the routes only so if I deleted it will restore again once new pvc is created.
2) In the documentaion they have mentioned more than (n-1) /2 .It means when more than 1 pod (atleast 2 if you have 3 etcd pods) will fail then automatically pods will try to recover. Recently in our QA env all the pods were down so it's better to implement disater recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2251245375
Hi @BadTorro try to enable the disaster recovery cron job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2254995575
@sudhir649, We are facing one more error in apisix. We use openid-connect plugin for authentication and authorization in the ApisixPluginConfig. When we try to hit the ingress of application, it gives 431 (Header too large) error. We tried removing few headers but it was breaking UI of application, so is there a way to solve this? Have you come across similar issue before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "gustysap (via GitHub)" <gi...@apache.org>.
gustysap commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2456146183
any solutions for this issue iam also facing the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "BadTorro (via GitHub)" <gi...@apache.org>.
BadTorro commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2252076245
@sudhir649 thanks for the tipp, need to verify - seems like I need an nfs storage provider to get the snapshot image to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "BadTorro (via GitHub)" <gi...@apache.org>.
BadTorro commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2251227436
Still having the same issue. We downloaded and added the entire chart dir, setting the etcd version in chart.yaml to "10.1.0" as suggested by @Lakshmi2k1
Are there any plans to have this fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "flea1lt (via GitHub)" <gi...@apache.org>.
flea1lt commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2154115657
There is one running etcd pod, run `etcdctl member list` after exec into the pod. And check the logs of crashed etcd pods, normally there was member id it used.
BTW, I think it's more likely a etcd problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 closed issue #11338: help request: Apisix ETCD going into Crash loop back off
URL: https://github.com/apache/apisix/issues/11338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2160575087
Apisix is working fine after upgrading the version of etcd in chart.yaml as "10.1.0". So, closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2255067405
@Thilip707
We are using the below configuration in apisix configmap as mentioned in docs
<img width="476" alt="image" src="https://github.com/user-attachments/assets/3358b2bf-115b-437d-a8dc-8bc854262788">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2157467535
> any solutions for this issue iam also facing the same issue for last 3 days
I have changed the etcd version in chart.yaml to "10.1.0". Now all pods are in running state. I'm checking few things in UI to make sure everything is working fine. If you are using Helm chart for deploying apisix, try this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Thilip707 (via GitHub)" <gi...@apache.org>.
Thilip707 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2157416571
any solutions for this issue iam also facing the same issue for last 3 days
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Thilip707 (via GitHub)" <gi...@apache.org>.
Thilip707 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2157477728
thanks will try and update mam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2153984151
> I think you need to try `etcdctl member list` first. This will help you verify if the member ID of the crashing pods matches the IDs from the etcdctl.
If it's cluster's etcd then we have to login into the node and execute the commands, since here it is running as pod, not sure where to execute etcdctl commands and also as the pods are in crash loop back off, I can't even exec into the pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan closed issue #11338: help request: Apisix ETCD going into Crash loop back off
URL: https://github.com/apache/apisix/issues/11338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3162166205
Based on all the feedback and "solutions" collected in the past, here is a summary:
1. The currently **primary** supported deployment methods for APISIX are Docker (VM) and Helm (Kubernetes).
2. APISIX has three deployment modes: Traditional (with ETCD), Decoupled (with ETCD) and Standalone (**without** ETCD). Refer to https://docs.api7.ai/apisix/production/deployment-modes
3. For deployment modes that require ETCD, we strongly **recommend deploying ETCD on a VM**. When deploying to Kubernetes, scaling ETCD pods may result in such issues mentioned above (this issue is unrelated to APISIX itself).
4. If you are deploying APISIX on Kubernetes, we recommend controlling the 'apache/apisix' gateway via the **APISIX Ingress Controller 2.0+** The new design does not use ETCD, so you will not encounter these issues again. Please read https://github.com/apache/apisix-ingress-controller/blob/master/docs/en/latest/upgrade-guide.md to understand the differences and upgrade steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3193944012
Hi @ch-gr, we already have a number of production users using this option. If there are no more additional bug reports within 6 weeks, we will consider making it generally available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "BadTorro (via GitHub)" <gi...@apache.org>.
BadTorro commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2259172299
> @sudhir649 @BadTorro Deploying new version of etcd worked for me initially but whenever the node is scaled down and scaled up, one of the etcd pods is going into crashloop. But since other two etcd pods were running, it wasn't affecting the route and upstream creations. Yet, we are about to use it in production environment, so wish there is a permanent fix. After reading the solution @sudhir649 you have pointed out, I have few questions. 1. Won't deleting the PVC cause loss of data that apisix needs. 2. For second solution, I feel it's good to give a try. (n-1)/2 , in my case number of etcd replicas is 3, so according to this even if one pod is crashing, the disaster recovery cron will run and take backup of pvc. But as you mentioned when two pods were crashing there was no change, when third pod also crashed then all three pods came to running state. But in my case only one or rarely two pods crashing. If you have any inputs let me know. Thanks in advance!
Regarding to that, I managed to get it work by basically:
- Deploying [longhorn](https://longhorn.io/) storage solution to the cluster
- Configured rancher desktop based on[ this guide](https://medium.com/@thizaom/rancher-desktop-with-rancher-longhorn-280e687f5022) to have open-iscsi in place and useable
- changed the storageclass in the dedicated etcd chart and related values.yaml file to "longhorn"
```
persistence:
enabled: true
storageClass: "longhorn"
```
- started everything with "tilt up"
Currently keeps on running and did not crash since.
However, we are now as well checking if the [Bitnami chart](https://artifacthub.io/packages/helm/bitnami/apisix) runs out of the box...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Lakshmi2k1 (via GitHub)" <gi...@apache.org>.
Lakshmi2k1 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2255063828
@sudhir649
we are using this configuration in apisix configmap as mentioned in docs
nginx_config: # config for render the template to genarate nginx.conf
http_server_configuration_snippet: |
large_client_header_buffers 4 32k;
http_end_configuration_snippet: |
reset_timedout_connection off;
client_header_buffer_size 1024k;
client_body_buffer_size 20m;
proxy_intercept_errors on;
large_client_header_buffers 8 128k;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "gustysap (via GitHub)" <gi...@apache.org>.
gustysap commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2462376162
Hi @Joeydelarago , when you delete the etcd, what data is gone ya? all route is gone or is still exist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3518885917
fyi for APISIX Ingress Controller 2.0.0 GA plan: https://github.com/apache/apisix-ingress-controller/issues/2514#issuecomment-3514640605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "serhiikucherenko (via GitHub)" <gi...@apache.org>.
serhiikucherenko commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2326281693
JFYI : `removeMemberOnContainerTermination:false` didn't help me on AKS cluster v.1.28.9
(docker.io/bitnami/etcd:3.5.10-debian-11-r2)
`{"level":"warn","ts":"2024-09-03T11:26:04.835351Z","caller":"etcdserver/server.go:1127","msg":"server error","error":"the member has been permanently removed from the cluster"}`
Any ideas? :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2277162820
> > @Lakshmi2k1
> >
> > 1. for deleting the pvc it's depends what data are you storing into it. In my case Or generally we are storing the routes only so if I deleted it will restore again once new pvc is created.
> > 2. In the documentaion they have mentioned more than (n-1) /2 .It means when more than 1 pod (atleast 2 if you have 3 etcd pods) will fail then automatically pods will try to recover. Recently in our QA env all the pods were down so it's better to implement disater recovery.
>
> I have enabled disaster recovery and deployed the helm chart, but this time not just etcd was crashing, the apisix pod stuck in init container, apisix ingress controller was crashing and the snapshot pod was also in error state. So, I rolled back to previous revision again after observing the pod status doesn't seem to change for a long time.
@Lakshmi2k1 The problem you encountered has nothing to do with disaster recovery. I have not experienced this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2251325397
Hi @BadTorro ,
I have found as of now two solution for it.
1) Intermittent solution for it delete the all three pvc and restart the pod.
2) check the readme.md file document in bitanami/etcd folder where they have explained how to enable the disaterrecovery cronjob.
In disaster recovery there is a cronjob it will take the back of pvc, if more than (n-1)/2 pods are failing then pods will automatically come back to running status with the help of backup pvc.
I have implememted the disaster recovery in my env, now I have seen that 2 pods are still failing and try to come back in running status , logs are also changed but unfortunately they are still not able to come back in running status.
But once the third pod got failed all the three pods are automatically come back to running status with the help of backup pvc.
So extract the etcd zip folder and in values. yaml enable the cronjob and zip it and redeploy it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "sudhir649 (via GitHub)" <gi...@apache.org>.
sudhir649 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2252197455
@BadTorro yes, tilt is working on local machine so you need to deploy nfs storage class.
By the way lakshmi solution won't work me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2151696993
Do you have a strong requirement to use `2.7.0`? I'm using the latest and pods are starting normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "Joeydelarago (via GitHub)" <gi...@apache.org>.
Joeydelarago commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-2461867969
I had this issue running on GKE. Uninstalling and deleting the etcd PVCs, then reinstalling fixed the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] help request: Apisix ETCD going into Crash loop back off [apisix]
Posted by "vichaos (via GitHub)" <gi...@apache.org>.
vichaos commented on issue #11338:
URL: https://github.com/apache/apisix/issues/11338#issuecomment-3402607443
@juzhiyuan [Custom Helm release names cause APISIX gateway readiness probe failures in standalone API-driven mode](https://github.com/apache/apisix-helm-chart/issues/882)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org