You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/04/14 13:16:49 UTC
[GitHub] [apisix] suninuni opened a new issue, #6845: help request: etcd OOM due to frequent deployments
suninuni opened a new issue, #6845:
URL: https://github.com/apache/apisix/issues/6845
### Description
The load of apisix-etcd deployed in the K8s cluster has been increasing, is this normal?
By rebuilding the etcd cluster, the memory dropped to 200Mi without any change for K8s cluster.
This phenomenon occurs in several of my frequently deployed clusters.
### Environment
- APISIX version (run `apisix version`): 2.11
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): 3.4.16
- APISIX ingress version: 1.4.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1099799471
The reason is that I found etcd oom, so I increased the memory for it, but it kept restoring from backup when it started. So, I can only rebuild etcd cluster.
So I guess that there are a lot of outdated information in etcd that have not been deleted. I haven't had time to find the relevant documentation on the data structures stored in etcd, is there anything I can query to confirm this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1100047973
> So I guess that there are a lot of outdated information in etcd that have not been deleted. I haven't had time to find the relevant documentation on the data structures stored in etcd, is there anything I can query to confirm this?
I can't found dirty data in etcd, but the memory usage is normal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1101008202
@suninuni ETCD supports MVCC, so each `PUT` operation will make it generate a new version for the key-value pair, you may enable the auto compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni closed issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
suninuni closed issue #6845: help request: etcd OOM due to frequent deployments
URL: https://github.com/apache/apisix/issues/6845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1099790162
> > due to frequent deployments
>
> How to understand this?
It means half of the upstreams (the specific numbers maybe 15 and each upstream have 2 pod) will be updated every 30 minutes.
> By the way, could you also check if you enable the server-info plugin?
Sorry, it's not being enabled before.
```
{"last_report_time":1649989004,"id":"67158603-3d3c-412b-896e-3da3cfe7763b","version":"2.11.0","hostname":"192-168-116-247","boot_time":1649989004,"etcd_version":"3.4.0","up_time":41}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1099767223
> due to frequent deployments
How to understand this?
By the way, could you also check if you enable the server-info plugin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #6845: help request: etcd OOM due to frequent deployments
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #6845:
URL: https://github.com/apache/apisix/issues/6845#issuecomment-1101346187
> @suninuni ETCD supports MVCC, so each `PUT` operation will make it generate a new version for the key-value pair, you may enable the auto compaction.
Thanks, enable compaction solved my problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org