You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/06/22 03:05:38 UTC
[GitHub] [apisix] nanamikon opened a new issue, #7301: help request: Load rebalance about websocket
nanamikon opened a new issue, #7301:
URL: https://github.com/apache/apisix/issues/7301
### Description
Hi all
We know that websocket will pass data through tcp after upgrade through http， so load balance by “**active tcp connection** should be most effective。
Refer to nginx doc about least_conn
`Specifies that a group should use a load balancing method where a request is passed to the server with the least number of active connections, taking into account weights of servers. If there are several such servers, they are tried in turn using a weighted round-robin balancing method.`
Active connection mean there is http request on the way, and I find it the same in the apisix and kong implement。 However， it is ineffective against websocket， beacause there is no http request after upgrade. Especially in the expansion situation, the new node can not get more connection than other nodes.
Any tips to implement a new loadbalance to recover this situation？
Many Thanks
### Environment
- APISIX version (run `apisix version`): 2.9
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1164372306
> Sorry for misleading you, the counter will be reduced in log phase The key question is
>
> ```
> the new node can not get more connection than other nodes
> ```
>
> When add a new node to upstream, it will call the function least_conn.new(), so binaryHeap will be rebuilded, all the counter will be lost....
Fair enough. That's indeed a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] nanamikon commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
nanamikon commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1163213414
Refer to least_conn.lua
```
get = function (ctx)
.......
info.score = info.score + info.effect_weight
servers_heap:update(server, info)
return server
end
```
```
after_balance = function (ctx, before_retry)
local server = ctx.balancer_server
local info = servers_heap:valueByPayload(server)
info.score = info.score - info.effect_weight
servers_heap:update(server, info)
......
```
I found that the counter will add after picking, and reduce in the log phase， so this will work only for the
http request of upgrade。 After ws upgrade， apisix will not kown how many active tcp connections now
In the product env， we find least_conn fall back to rr through our monitor dashboard。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1163819566
> > After ws upgrade， apisix will not kown how many active tcp connections now
>
> if http upgrade to ws, the counter won't be reduced in log phase?
I don't think so, log phase can still be run when the WebSocket connection is closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1165228508
Sorry, now I can get your point.
However, when a new node is added, the upstream conf_version will increase, and require to create a new balancer due to lrucache miss:
https://github.com/apache/apisix/blob/8e1a1fd10ab55891f63f15f0ded3df88ad6f44bd/apisix/upstream.lua#L290-L294
https://github.com/apache/apisix/blob/8e1a1fd10ab55891f63f15f0ded3df88ad6f44bd/apisix/balancer.lua#L240
Currently, there is no way to get the previous upstream conf and the its cached data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] nanamikon commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
nanamikon commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1164148844
Sorry for misleading you, the counter will be reduced in log phase
The key question is
···
the new node can not get more connection than other nodes
···
When add a new node, it will call the function least_conn.new(), so binaryHeap will be rebuilded, all the counter will be lost.
So the new node can not get more connection than other nodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] closed issue #7301: help request: Load rebalance about websocket
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7301: help request: Load rebalance about websocket
URL: https://github.com/apache/apisix/issues/7301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] nanamikon commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
nanamikon commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1165136744
```
The new node can get more connection than other nodes
```
It is important in the following issues
1. auto scale base on cpu uasge
2. small size pod， 2 cpu ， only support 20 ~ 30 connections，
Refer to least_conn.lua
```
for server, weight in pairs(up_nodes) do
**local score = 1 / weight**
-- Note: the argument order of insert is different from others
servers_heap:insert({
server = server,
effect_weight = 1 / weight,
score = score,
}, server)
end
```
Improment
1、We can build a new map to store node -> connection count, and init servers_heap with the info of this map
2、We should call servers_heap:insert in the random order just as rr， maybe we can start at the random index of up_nodes, and insert into it one by one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1162711227
> specially in the expansion situation, the new node can not get more connection than other nodes
Maybe I am missing something. The new node will get more connections because the old nodes already have some connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1163296088
> After ws upgrade， apisix will not kown how many active tcp connections now
if http upgrade to ws, the counter won't be reduced in log phase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] nanamikon commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
nanamikon commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1164007195
> > After ws upgrade， apisix will not kown how many active tcp connections now
>
> if http upgrade to ws, the counter won't be reduced in log phase?
1、client ---- http upgrade request ----> apisix
2、apisix -----http upgrade request ----> upstream
3、upstream -----support upgrade, return 101 ----> apisix
4、apisix -----support upgrade, return 101 ----> client
5、client ----- pass data through tcp channel ----> apisix
6、client ----- forward data through tcp channel ----> upstream
The counter will be added in the phase 1， and be reduced in the phase 4。 After upgrade, the counter is 0 now ， however, there is still an active tcp connection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7301: help request: Load rebalance about websocket
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1162711840
> specially in the expansion situation, the new node can not get more connection than other nodes
Maybe I am missing something. The new node will get more connections because the old nodes already have some connections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7301: help request: Load rebalance about websocket
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1584314706
This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] github-actions[bot] commented on issue #7301: help request: Load rebalance about websocket
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7301:
URL: https://github.com/apache/apisix/issues/7301#issuecomment-1604044857
This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org