You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Richard-lululemon (via GitHub)" <gi...@apache.org> on 2025/07/28 09:15:04 UTC

[I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Richard-lululemon opened a new issue, #12466:
URL: https://github.com/apache/apisix/issues/12466

   ### Description
   
   we have 40k+routes and 20k+ ups, when routes updates(400+ each time), cpu can up to 100% for 3 minutes. is this situation normal? how many routes can apisix support?
   
   ### Environment
   
   - APISIX version (run `apisix version`): 3.9.0
   - Operating system (run `uname -a`):
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Richard-lululemon (via GitHub)" <gi...@apache.org>.
Richard-lululemon closed issue #12466: help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups
URL: https://github.com/apache/apisix/issues/12466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Richard-lululemon (via GitHub)" <gi...@apache.org>.
Richard-lululemon commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3126389354

   > Can you share your server information and how you deployed it?
   
   we deploy in k8s,has more than 15 workers,enable k8s discovery
   cpu info: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHZ


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Richard-lululemon (via GitHub)" <gi...@apache.org>.
Richard-lululemon commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3130491119

   > 1 pod 15 workers?
   > 
   > we have 50K+ routes, and met the same situation, the worker will rebuild the radixtree every time it watched a change. and at that time, the incoming requests would stuck to wait for the rebuilding [#10268](https://github.com/apache/apisix/issues/10268)
   > 
   > * so, use only 4 workers for each pod
   > * 4 workers don't rebuild the tree at the same time
   > * etcd compatcion [feat: I hope to provide best practices for optimizing Etcd as it serves as the core database of Apisix #9946 (comment)](https://github.com/apache/apisix/issues/9946#issuecomment-1670542476)
   
   How did you solve your problem?
   should i use https://github.com/TencentBlueKing/blueking-apigateway-apisix/blob/master/src/build/patches/004_radixtree_uri_with_parameter_rebuild_with_interval.patch ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Richard-lululemon (via GitHub)" <gi...@apache.org>.
Richard-lululemon commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3134771894

   > the patch works for me. (we use radixtree_uri_with_parameter, so patch the file, you should check your config)
   > 
   > if your rebuilding costs more than 15s, maybe you should change `random(5, 15)` to a bigger value.
   > 
   > You can choose one pod, modify the codes, then do `apisix reload`, and trigger the updates to see the difference.
   
   it works well,thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3130551426

   the patch works for me.
   
   if your rebuilding costs more than 15s, maybe you should change `random(5, 15)` to a bigger value.
   
   You can choose one pod, modify the codes, then do `apisix reload`, and trigger the updates to see the difference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3127291122

   1 pod 15 workers? 
   
   -----
   
   we have 50K+ routes, and met the same situation, the worker will rebuild the radixtree every time it watched a change. and at that time, the incoming requests would stuck to wait for the rebuilding #10268
   
   - so, use only 4 workers for each pod
   - 4 workers don't rebuild the tree at the same time
   - etcd compatcion https://github.com/apache/apisix/issues/9946#issuecomment-1670542476
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3126343790

   Can you share your server information and how you deployed it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "Richard-lululemon (via GitHub)" <gi...@apache.org>.
Richard-lululemon commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3130484198

   > 1 pod 15 workers?
   > 
   > we have 50K+ routes, and met the same situation, the worker will rebuild the radixtree every time it watched a change. and at that time, the incoming requests would stuck to wait for the rebuilding [#10268](https://github.com/apache/apisix/issues/10268)
   > 
   > * so, use only 4 workers for each pod
   > * 4 workers don't rebuild the tree at the same time
   > * etcd compatcion [feat: I hope to provide best practices for optimizing Etcd as it serves as the core database of Apisix #9946 (comment)](https://github.com/apache/apisix/issues/9946#issuecomment-1670542476)
   
   1 pod 2 worker, our k8s cluster has more than 15 nodes.
   when i set 4 workers, it seems 4 workers rebuild the tree at the same time
   
   <img width="574" height="85" alt="Image" src="https://github.com/user-attachments/assets/4f2960df-41b6-46ef-bdab-485828b29bc0" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] help request: cpu too high for 3 mins when routes update, we have 40k+routes and 20k+ ups [apisix]

Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #12466:
URL: https://github.com/apache/apisix/issues/12466#issuecomment-3217033668

   Thanks @wklken @Richard-lululemon for pointing out the patches https://github.com/TencentBlueKing/blueking-apigateway-apisix/tree/master/src/build/patches. cc @moonming @bzp2010 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org