You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Tanchao1997 (via GitHub)" <gi...@apache.org> on 2023/10/09 08:58:07 UTC
[I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Tanchao1997 opened a new issue, #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001
### Issue description
I have 1000+ ApisixUpstream and 2000+ ApisixRoute, when new leader elected, it costs about 10 minutes for a updated ApisixUpstream to be synced. I think it's probably because of too many events need to be dealed with. Is there any way to resolve this?
### Environment
- your apisix-ingress-controller version :1.6.1
- your apisix version : 2.15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "pawankukreja01 (via GitHub)" <gi...@apache.org>.
pawankukreja01 commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1794262197
You are experiencing a delay in syncing your ApisixUpstream and ApisixRoute when a new leader is elected. This delay can be attributed to the large number of events that need to be processed. Unfortunately, there is no one-size-fits-all solution to this problem.
**Increase the number of replicas**: You can increase the number of replicas of your ApisixUpstream and ApisixRoute to reduce the load on each replica. This will help to distribute the load more evenly and reduce the time it takes to sync the data.
**Reduce the number of events**: You can try to reduce the number of events that need to be processed by your ApisixUpstream and ApisixRoute. This can be achieved by optimizing your code or by reducing the number of updates that are made to your ApisixUpstream and ApisixRoute.
**Use a different approach**: You can try using a different approach to sync your ApisixUpstream and ApisixRoute. For example, you can use a different tool or library that is better suited to handle large volumes of data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #2001: request help: when new leader elected, it cost several minutes for a CR to be synced
URL: https://github.com/apache/apisix-ingress-controller/issues/2001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1984881622
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "grootTwig (via GitHub)" <gi...@apache.org>.
grootTwig commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1755128277
> Can you try the latest version? Also, please provide the configuration file used by the APISIX Ingress controller.
Sure, here is the configuration file
------------
log_level: "info"
log_output: "/app/logs/ingress-controller/ingress-controller.log"
log_rotate_output_path: ""
log_rotation_max_size: 100
log_rotation_max_age: 0
log_rotation_max_backups: 0
cert_file: "/etc/webhook/certs/cert.pem"
key_file: "/etc/webhook/certs/key.pem"
http_listen: ":8080"
https_listen: ":8443"
ingress_publish_service: ""
ingress_status_address: []
enable_profiling: true
apisix-resource-sync-interval: "0s"
# Kubernetes related configurations.
kubernetes:
kubeconfig: ""
resync_interval: "6h"
namespace_selector: ["apisix-ingress-controller-name=gateway-ingress-controller-dev"]
election_id: "gateway-ingress-controller-leader"
ingress_class: "gateway"
ingress_version: "networking/v1"
watch_endpointslices: false
apisix_route_version: "apisix.apache.org/v2"
enable_gateway_api: false
api_version: apisix.apache.org/v2
plugin_metadata_cm: plugin-metadata-config-map
# APISIX related configurations.
apisix:
admin_api_version: v2
default_cluster_base_url: "http://gateway-admin.gateway:9180/apisix/admin"
default_cluster_admin_key: ""
default_cluster_name: "gateway"
------------
I'll try the latest version later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "grootTwig (via GitHub)" <gi...@apache.org>.
grootTwig commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1797432623
> You are experiencing a delay in syncing your ApisixUpstream and ApisixRoute when a new leader is elected. This delay can be attributed to the large number of events that need to be processed. Unfortunately, there is no one-size-fits-all solution to this problem.
>
> **Increase the number of replicas**: You can increase the number of replicas of your ApisixUpstream and ApisixRoute to reduce the load on each replica. This will help to distribute the load more evenly and reduce the time it takes to sync the data.
>
> **Reduce the number of events**: You can try to reduce the number of events that need to be processed by your ApisixUpstream and ApisixRoute. This can be achieved by optimizing your code or by reducing the number of updates that are made to your ApisixUpstream and ApisixRoute.
>
> **Use a different approach**: You can try using a different approach to sync your ApisixUpstream and ApisixRoute. For example, you can use a different tool or library that is better suited to handle large volumes of data.
Thanks, i've resolved this delay by adding comparision to the local cache when synchronizing CR to apisix and CR status to k8s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "grootTwig (via GitHub)" <gi...@apache.org>.
grootTwig commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1755136713
Sure, here is the configuration file:
-----------
log_level: "info"
log_output: "/app/logs/ingress-controller/ingress-controller.log"
log_rotate_output_path: ""
log_rotation_max_size: 100
log_rotation_max_age: 0
log_rotation_max_backups: 0
cert_file: "/etc/webhook/certs/cert.pem"
key_file: "/etc/webhook/certs/key.pem"
http_listen: ":8080"
https_listen: ":8443"
ingress_publish_service: ""
ingress_status_address: []
enable_profiling: true
apisix-resource-sync-interval: "0s"
kubernetes:
kubeconfig: ""
resync_interval: "6h"
namespace_selector: ["apisix-ingress-controller-name=sn-gateway-ingress-controller-dev"]
election_id: "sn-gateway-ingress-controller-leader"
ingress_class: "sn-gateway"
ingress_version: "networking/v1"
watch_endpointslices: false
apisix_route_version: "apisix.apache.org/v2"
enable_gateway_api: false
api_version: apisix.apache.org/v2
plugin_metadata_cm: plugin-metadata-config-map
apisix:
admin_api_version: v2
default_cluster_base_url: "http://sn-gateway-admin.gateway-dev:9180/apisix/admin"
default_cluster_admin_key: "edd1c9f034335f136f87ad84b625c8f1"
default_cluster_name: "sn-gateway"
-------------
I'll try the latest version later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1928604040
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "tao12345666333 (via GitHub)" <gi...@apache.org>.
tao12345666333 commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1754549322
Can you try the latest version? Also, please provide the configuration file used by the APISIX Ingress controller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] request help: when new leader elected, it cost several minutes for a CR to be synced [apisix-ingress-controller]
Posted by "grootTwig (via GitHub)" <gi...@apache.org>.
grootTwig commented on issue #2001:
URL: https://github.com/apache/apisix-ingress-controller/issues/2001#issuecomment-1757126266
> Can you try the latest version? Also, please provide the configuration file used by the APISIX Ingress controller.
I've tried the latest version, it still costed 10+ minutes for 2000+ ApisixRoute and 1000+ ApisixUpstream to be synced. I added some metrics to observe the size of workqueue, and here is the trend chart.
<img width="1397" alt="image" src="https://github.com/apache/apisix-ingress-controller/assets/37180993/30e07de7-d418-440e-b14f-05f6a6fdf195">
I've noticed that comparison in resource sync is supported in v1.7.0 (https://github.com/apache/apisix-ingress-controller/commit/2cb99b89b99d5e8ce1dd277a0868c177630b4f43), it will accelerate the process if we apply it to processing add/update/delete events, is there any reason why we can't do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org