You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/05/13 06:30:45 UTC
[GitHub] [apisix] redynasc opened a new issue, #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
redynasc opened a new issue, #7042:
URL: https://github.com/apache/apisix/issues/7042
### Current Behavior
Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
### Expected Behavior
smooth memory usage
### Error Logs
Memory information of the production environment, captured using the openresty xray tool
### Steps to Reproduce
1. change worker_processes to 1 for easy observation
2. create upstream
curl "127.0.0.1:9080/apisix/admin/upstreams" -XPOST -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
{
"type": "chash",
"key": "chash-key",
"retries": 0,
"name": "stress_apisix-memory-test_28002",
"desc": "stress_apisix-memory-test_28002",
"hash_on": "header",
"timeout": {
"send": 0.1,
"connect": 0.1,
"read": 0.1
},
"nodes": [
{
"port": 28002,
"weight": 5000,
"host": "192.168.0.1"
},
{
"port": 28002,
"weight": 5000,
"host": "192.168.0.2"
}
]
}
'
3. create route
curl "127.0.0.1:9080/apisix/admin/routes" -XPOST -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
{
"upstream_id": "**the upstream id created in step 2**",
"hosts": [
"sc.test.com"
],
"uris": [
"/mem/test"
],
"name": "duisys>apisix-memory-test>prod",
"desc": "duisys>apisix-memory-test>prod"
}
'
4. run below command
for((i=1;i<100;i++)); do
curl "127.0.0.1:9080/apisix/admin/upstreams/{**the upstream id created in step 2**}" -XPATCH -H"x_api_key:edd1c9f034335f136f87ad84b625c8f1" -d '
{
"desc": "stress_apisix-memory-test_28002"
}
' > /dev/null 2>&1 ;
curl "127.0.0.1:9080/mem/test" -H"host:sc.test.com" > /dev/null 2>&1;
done
5. the memory cost of worker is up to 1.2GB
### Environment
- APISIX version :2.13.1
- Operating system:Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- OpenResty:
nginx version: openresty/1.19.9.1
built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
built with OpenSSL 1.1.1n 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.19.9.1.5 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.20 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../ngx_stream_lua-0.0.10 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-ap
i/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../mod_dubbo --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../ngx_multi_upstream_module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../apisix-nginx-module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../apisix-nginx-module/src/stream --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../wasm-nginx-module --add-module=/tmp/tmp.HOhTx9UT8p/openresty-1.19.9.1/../lua-var-nginx-module --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_modul
e --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
- etcd version: 3.5.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
URL: https://github.com/apache/apisix/issues/7042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1128297659
After a second thought, I have changed my mind.
As the 160 is widely used in several implementations:
https://github.com/ultrabug/uhashring
https://www.nginx.com/resources/wiki/modules/consistent_hash/
I think we should just gcd the weights and spend the remind effort on another chash implementation like Maglev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1128355848
> I think we should just gcd the weights and spend the remind effort on another chash implementation like Maglev.
My thoughts are similar. The current chash implementation doesn't seem to be able to avoid having too many `npoints` (e.g. two extreme primes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1128291621
> > 2. find a multiple which can make the min weight >= 1000, instead of a hard-code 1000 multiple
>
> Not quite sure where this 1000 comes from, please help me to point it out.
Sorry, it should be 160. Fixed in the original issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] redynasc commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
redynasc commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1127101473
In our production environment, the first solution was used temporarily
It seems that the second one is a better general solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] redynasc commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
redynasc commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1125702570
## my analysis
+ in apisix 2.13.1
1. the weight sum of all nodes in the upstream is 10000
2. rest.chash creates a chash table that contains 1600000(10000*CONSISTENT POINTS) point structures, each of which is 8 bytes in size. The total size of the chash table is at least 12.20MB
3. the chash table is cached in the lrucache queue, the cache key is **the address **of the upstream object
4. When the upstream is modified through the admin interface, a new upstream will be created, and a new chash table will be created and put into the lrucache queue (which is a new key), but the old chash table is still in the lrucache queue, not is released immediately until the lrucache queue is full
5. caused a memory leak
+ in apisix 1.15
1. In version 1.15 there is no such problem
2. This is because the key of the chash table in the lrucache queue is the upstream id. When the upstream is modified, the upstream id will not change. The new chash table will overwrite the old one in the lrucache queue, and the old one will GC dropped
+ Why in version 2.X, the lrucache key uses the upstream address？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1126930010
> Why in version 2.X, the lrucache key uses the upstream address？
It is because we add upstream discovery so the same upstream id may have different upstream addresses.
Some solutions to this problem (sorted by my favorite):
1. dividing the weight by a GCD before using it. Hoping the weights are not prime.
2. use a different chash algorithm, like Nginx's / Maglev / AnchorHash
3. don't use LRU for the upstream objects. This requires refactor at high risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1127219670
We can optimize lua-resty-balancer with two steps:
1. use GCD to reduce total weight
2. find a multiple which can make the min weight >= 1000, instead of a hard-code 1000 multiple
As for different chash algorithms, since different algorithms have Cons & Pros, we will leave it to a separate topic but not as a bugfix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7042: bug: Obvious memory leak when the weight sum of all nodes in the upstream of the chash type is large
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7042:
URL: https://github.com/apache/apisix/issues/7042#issuecomment-1127493781
> 2\. find a multiple which can make the min weight >= 1000, instead of a hard-code 1000 multiple
Not quite sure where this 1000 comes from, please help me to point it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org