You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Nobilta (via GitHub)" <gi...@apache.org> on 2023/10/17 03:56:12 UTC
[I] bug: APISIX3.5 has high memory usage problems [apisix]
Nobilta opened a new issue, #10349:
URL: https://github.com/apache/apisix/issues/10349
### Current Behavior
Since using APISIX3.5, the memory has been growing continuously, as shown in the figure below.

Using the slabtab command, you can see that radixtree occupies a high amount of space.

There is no such problem on APISIX2.9 or APISIX3.3.
### Expected Behavior
_No response_
### Error Logs
_No response_
### Steps to Reproduce
Freshly install and run APISIX3.5,then use ETCD as data surface.
### Environment
- APISIX version (run `apisix version`):3.5
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):openresty/1.21.4.1
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "TakiJoe (via GitHub)" <gi...@apache.org>.
TakiJoe commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1786897581
> Combining your information, I very much suspect that the problem was introduced in 3.4; combined with the changelog, I guess it is caused by the modification of etcd. Can you help me test it? (I tried to reproduce it on centos but failed. )
>
> Use the 3.3 etcd file to replace the 3.5 one. Take a look at wget https://raw.githubusercontent.com/apache/apisix/release/3.3/apisix/core/config_etcd.lua
OK, I'll try to reproduce & test it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1773641347
@Sn0rt thanks for the detailed information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "TakiJoe (via GitHub)" <gi...@apache.org>.
TakiJoe commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1870082200
I did a grayscale test in the production environment and it looks fine so far.(apisix3.5 patched with this two fix https://github.com/apache/apisix/pull/10614
https://github.com/apache/apisix/pull/10671 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 closed issue #10349: bug: APISIX3.5 has high memory usage problems
URL: https://github.com/apache/apisix/issues/10349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1846788250
We have located this problem: https://github.com/apache/apisix/pull/10614, it will affect all versions between 3.4.0 and 3.7.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "Nobilta (via GitHub)" <gi...@apache.org>.
Nobilta commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1776403398
> Stage information:
>
> 1. From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.
> 2. APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.
> 3. APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.
> 4. User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.
>
> further processing
>
> 1. Wait for the user to switch to tlinux and retest.
After upgrading the kernel to 5.4.119-19.0009.32, the phenomenon is the same as before. The openresty worker memory keeps growing after receiving traffic. In the same environment as APISIX 3.3, using the same plug-in configuration (different data plane), version 3.5 has a memory leak problem, but version 3.3 works normally.
<img width="1817" alt="357c914aafdb1ed9d33a33ebf0243e6" src="https://github.com/apache/apisix/assets/59631965/2dd0aca8-5a03-442e-8727-f93f296b2413">

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1772037336
Stage information:
1. From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.
2. APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.
3. APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.
4. User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.
further processing
1. Wait for the user to switch to tlinux and retest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1792874098
@deluxor we are tracking this problem, do you have any more clues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1869863832
This commit is needed too: https://github.com/apache/apisix/commit/0a85ba4a07fdd58ec01f8deaed568c833fd3717f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "TakiJoe (via GitHub)" <gi...@apache.org>.
TakiJoe commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1818177849
Here is the captured flame graphs and memory usage.






--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "zsmlinux (via GitHub)" <gi...@apache.org>.
zsmlinux commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-2440432908
> Here is the captured flame graphs and memory usage. 
>
> 
>
> 
>
> 
>
> 
>
> 
Excute me, could you tell me the steps for how to generate memory flame graph?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "monkeyDluffy6017 (via GitHub)" <gi...@apache.org>.
monkeyDluffy6017 commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1851394401
I will close this proble since it has been fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "deluxor (via GitHub)" <gi...@apache.org>.
deluxor commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1792865438
<img width="1019" alt="Screenshot 2023-11-03 at 17 34 09" src="https://github.com/apache/apisix/assets/5440296/8977e316-68b6-4a1a-9b31-be6b785830da">
We also spotted the same issue with the recent update of version 3.5;
Kept rising until OOM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "Nobilta (via GitHub)" <gi...@apache.org>.
Nobilta commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1776521774
> > Stage information:
> >
> > 1. From the current perspective of slab leaks in Linux, it cannot be concluded that APISIX also has memory leaks.
> > 2. APISIX's radixtree-based route matching is completely in user mode and has nothing to do with the kernel data structure radix_tree_node.
> > 3. APISIX is implemented based on NGX. Because ngx has its own memory management, from the perspective of the operating system, the ngx memory has not been released.
> > 4. User feedback The reason why 3.3 APISIX is ok is that the operating system kernel is different. 3.3 runs on the tlinux kernel.
> >
> > further processing
> >
> > 1. Wait for the user to switch to tlinux and retest.
>
> After upgrading the kernel to 5.4.119-19.0009.32, the phenomenon is the same as before. The openresty worker memory keeps growing after receiving traffic. In the same environment as APISIX 3.3, using the same plug-in configuration (different data plane), version 3.5 has a memory leak problem, but version 3.3 works normally. <img alt="357c914aafdb1ed9d33a33ebf0243e6" width="1817" src="https://user-images.githubusercontent.com/59631965/277531823-2dd0aca8-5a03-442e-8727-f93f296b2413.png"> 
According to the memory data of the last dump, the largest address space occupied after string processing is the upstream node information related to the use of the traffic-split plugin.The upstream node information that appears in the dump file is all part 1 in the figure, and the node 2 almost does not exist in the dump file. so we suspect whether it is related to the modification of the traffic-split plugin in version 3.5.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1786893036
Combining your information, I very much suspect that the problem was introduced in 3.4; combined with the changelog, I guess it is caused by the modification of etcd. Can you help me test it? (I tried to reproduce it on centos but failed. )
Use the 3.3 etcd file to replace the 3.5 one. Take a look at wget https://raw.githubusercontent.com/apache/apisix/release/3.3/apisix/core/config_etcd.lua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1766136964
thx, you Report. pls assgin to me .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: APISIX3.5 has high memory usage problems [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on issue #10349:
URL: https://github.com/apache/apisix/issues/10349#issuecomment-1765652038
cc: @kingluo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org