You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/01/06 14:36:12 UTC
[GitHub] [apisix] JiaYueHuang opened a new issue #6034: bug: problems caused by stream_proxy (tcp proxy)
JiaYueHuang opened a new issue #6034:
URL: https://github.com/apache/apisix/issues/6034
### Issue description
My apisix configuration file (config.yaml) shows below (only shows the problem related part):
I use the stream_proxy (tcp proxy) to listen ``vip`` (provided by keepalived) and 11211, 3306 ports (related to memcached and mysql services), and then I will create ``upstream`` and ``route`` related to memcached and mysql services to achive High Availability
[Uploading Description of problems caused by stream_proxy (tcp proxy).pdf…]()
### Environment
- apisix version (cmd: `apisix version`): 2.10.0
- OS (cmd: `uname -a`): centos 7.6
- OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
- etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API): 3.4.3
- apisix-dashboard version, if have:
- the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
- luarocks version, if the issue is about installation (cmd: `luarocks --version`):
### Steps to reproduce
[Uploading Description of problems caused by stream_proxy (tcp proxy).pdf…]()
### Actual result
see the file below
[Description of problems caused by stream_proxy (tcp proxy).pdf](https://github.com/apache/apisix/files/7822488/Description.of.problems.caused.by.stream_proxy.tcp.proxy.pdf)
### Error log
[Uploading Description of problems caused by stream_proxy (tcp proxy).pdf…]()
### Expected result
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008568780
> 1. Does the `stream_proxy` (tcp proxy) is a short connection?
AFAIK, not short connection by default. pls show the stream proxy configuration in `nginx.conf`.
This error, from your description, APISIX disconnected connection? cc @spacewander the stream proxy has timeout configuration in the upstream, I think it has something to do with the timeout?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1011055040
> > 1. Does the `stream_proxy` (tcp proxy) is a short connection?
>
> AFAIK, not short connection by default. pls show the stream proxy configuration in `nginx.conf`.
>
> This error, from your description, APISIX disconnected connection? cc @spacewander the stream proxy has timeout configuration in the upstream, I think it has something to do with the timeout?
configuration sample:
```
apisix:
node_listen:
- ip: 176.76.8.246
port: 9080
- ip: 176.76.8.246
port: 35357
- ip: 176.76.8.246
port: 5000
- ip: 176.76.8.246
port: 9292
- ip: 176.76.8.246
port: 9191
- ip: 176.76.8.246
port: 8776
- ip: 176.76.8.246
port: 8774
- ip: 176.76.8.246
port: 9696
- ip: 176.76.8.246
port: 8786
- ip: 176.76.8.246
port: 8778
- ip: 176.76.8.246
port: 8777
- ip: 176.76.8.246
port: 6666
- ip: 176.76.8.246
port: 8780
- ip: 176.76.8.246
port: 8779
enable_tcp_pp: true
enable_tcp_pp_to_upstream: true
stream_proxy:
only: false
tcp:
- "176.76.8.246:3306"
- "176.76.8.246:11211"
allow_admin:
#- 127.0.0.0/24
#- 0.0.0.0/24
- 0.0.0.0/0
admin_key:
- name: admin
key: edd1c9f034335f136f87ad84b625c8f1
role: admin
router:
http: radixtree_uri_with_parameter
etcd:
host:
- "http://176.76.8.165:2379"
- "http://176.76.8.181:2379"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008485771
I use apisix to encapsulate the openstack’s API to achieve high availability and rate limiting.
1.Problem description
My apisix configuration file (config.yaml) shows below (only shows the problem related part):
I use the stream_proxy (tcp proxy) to listen ``vip`` (provided by keepalived) and 11211, 3306 ports (related to memcached and mysql services), and then I will create ``upstream`` and ``route`` related to memcached and mysql services to achive High Availability (shown below).
Notice: I observed that, for msyql and memcache services, stream_proxy (tcp proxy) must be used, otherwise (if still use the http proxy), mysql and memcache services wouldn’t work out (commands below couldn’t connect correctly).
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang edited a comment on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang edited a comment on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1011055040
> > 1. Does the `stream_proxy` (tcp proxy) is a short connection?
>
> AFAIK, not short connection by default. pls show the stream proxy configuration in `nginx.conf`.
>
> This error, from your description, APISIX disconnected connection? cc @spacewander the stream proxy has timeout configuration in the upstream, I think it has something to do with the timeout?
shown below:
```
stream {
lua_package_path "$prefix/deps/share/lua/5.1/?.lua;$prefix/deps/share/lua/5.1/?/init.lua;/usr/local/apisix/?.lua;/usr/local/apisix/?/init.lua;;/usr/local/apisix/?.lua;./?.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.lua;/usr/lib64/lua/5.1/?.lua;/usr/lib64/lua/5.1/?/init.lua;";
lua_package_cpath "$prefix/deps/lib64/lua/5.1/?.so;$prefix/deps/lib/lua/5.1/?.so;;./?.so;/usr/lib64/lua/5.1/?.so;/usr/lib64/lua/5.1/loadall.so;";
lua_socket_log_errors off;
lua_max_pending_timers 16384;
lua_max_running_timers 4096;
lua_shared_dict lrucache-lock-stream 10m;
lua_shared_dict plugin-limit-conn-stream 10m;
lua_shared_dict etcd-cluster-health-check-stream 10m;
resolver 8.8.8.8 114.114.114.114;
resolver_timeout 5;
# stream configuration snippet starts
# stream configuration snippet ends
upstream apisix_backend {
server 127.0.0.1:80;
balancer_by_lua_block {
apisix.stream_balancer_phase()
}
}
init_by_lua_block {
require "resty.core"
apisix = require("apisix")
local dns_resolver = { "8.8.8.8", "114.114.114.114", }
local args = {
dns_resolver = dns_resolver,
}
apisix.stream_init(args)
}
init_worker_by_lua_block {
apisix.stream_init_worker()
}
server {
listen xxx.xx.8.246:3306 reuseport;
listen xxx.xx.8.246:11211 reuseport;
preread_by_lua_block {
apisix.stream_preread_phase()
}
proxy_pass apisix_backend;
log_by_lua_block {
apisix.stream_log_phase()
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang removed a comment on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang removed a comment on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008485771
I use apisix to encapsulate the openstack’s API to achieve high availability and rate limiting.
### Problem description
My apisix configuration file (config.yaml) shows below (only shows the problem related part):
I use the stream_proxy (tcp proxy) to listen ``vip`` (provided by keepalived) and 11211, 3306 ports (related to memcached and mysql services), and then I will create ``upstream`` and ``route`` related to memcached and mysql services to achive High Availability (shown below).
Notice: I observed that, for msyql and memcache services, stream_proxy (tcp proxy) must be used, otherwise (if still use the http proxy), mysql and memcache services wouldn’t work out (commands below couldn’t connect correctly).
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
For simplicity, related upstreams has only one node:
Mysql’s upstream shows blow:
Memcached’s upstream shows below:
Mysql’s stream_route (experiment shows that it’s also valid for ordinary route, but will have the same problem) shows below (matched by 3306 port and forward the service to specific node):
Memcache’s stream_route :
Then we test the mysql and memcache services, execute the command below (it shows correctly):
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
### 1) Problems caused by nova-api (memcached related)
Configuration related to memcached in nova.conf (if use the ``vip``, there are some problems in nova-api.log, if use specific ip, no ERROR will occur):
### 2) Problems caused by masakari (mysql related)
I alse use the ``vip`` to connect mysql in masakari configuration file (if use real ip, no ERROR will occur):
### Summary
Because I only set one node in upstream, and evidence shows that apisix is functioning correctly, but it still cause some minor problem, which seems related to stream_proxy (tcp proxy), and belows are my question about the apisix’s stream_proxy:
1.Does the stream_proxy (tcp proxy) is a short connection? And if so, it will interrupt in the client side, which will cause the server’s side to trigger some error log (refer to: https://www.jb51.net/article/57602.htm).
2.Is there any alternative solution to connect mysql and memcached with apisix?
If I use the haproxy to achieve High Availability, problems above will not occur (but haproxy doesn’t have rate limiting functionality).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008725776
Look like you have set the upstream's timeout to 30s, which is relatively short for TCP connection. Does it cause the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang edited a comment on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang edited a comment on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008485771
I use apisix to encapsulate the openstack’s API to achieve high availability and rate limiting.
1.Problem description
My apisix configuration file (config.yaml) shows below (only shows the problem related part):
I use the stream_proxy (tcp proxy) to listen ``vip`` (provided by keepalived) and 11211, 3306 ports (related to memcached and mysql services), and then I will create ``upstream`` and ``route`` related to memcached and mysql services to achive High Availability (shown below).
Notice: I observed that, for msyql and memcache services, stream_proxy (tcp proxy) must be used, otherwise (if still use the http proxy), mysql and memcache services wouldn’t work out (commands below couldn’t connect correctly).
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
For simplicity, related upstreams has only one node:
Mysql’s upstream shows blow:
Memcached’s upstream shows below:
Mysql’s stream_route (experiment shows that it’s also valid for ordinary route, but will have the same problem) shows below (matched by 3306 port and forward the service to specific node):
Memcache’s stream_route :
Then we test the mysql and memcache services, execute the command below (it shows correctly):
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1079656214
I guess the key-point was on `vip`, please check the `arp_ignore` `arp_announce` and `rp_filter`.
Under using `keepalived` aside with `IPVS`, focus on `ARP` first, see details [here](http://kb.linuxvirtualserver.org/wiki/Using_arp_announce/arp_ignore_to_disable_ARP).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1011057678
> Look like you have set the upstream's timeout to 30s, which is relatively short for TCP connection. Does it cause the problem?
My upstream's parameters are shown below:
```
function create_upstream {
upstream_id=$1
req_body=$2
#jy: 去除请求体中的空格
req_body=`echo ${req_body} | sed 's/ //g'`
#echo ${req_body}
curl http://${apisix_ip}:${apisix_port}/apisix/admin/upstreams/${upstream_id} -H "X-API-KEY: ${api_key}" -X PUT -d ''${req_body}''
}
create_upstream ${upstream_id} '{
"type":"roundrobin",
"nodes":'${node_info}',
"timeout": {
"connect": 15,
"send": 15,
"read": 15
},
"checks": {
"active": {
"http_path": "/",
"healthy": {
"interval": 2,
"successes": 2
},
"unhealthy": {
"interval": 2,
"http_failures": 3
}
}
},
"keepalive_pool.size": 100,
"keepalive_pool.idle_timeout": 361,
"keepalive_pool.requests": 10000,
"name": "'${upstream_name}'",
"desc": "'${upstream_desc}'"
}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1011803173
the `timeout` in upstream is the timeout for connecting, sending and receiving messages.
the `keepalive_pool ` the upstream to have its separate connection pool. Its children fields, like requests, can be used to configure the upstream keepalive options.
They are different configurations.
I suggest you debug the timeout configuration first to verify that the problem is timeout related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang edited a comment on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang edited a comment on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1011057678
> Look like you have set the upstream's timeout to 30s, which is relatively short for TCP connection. Does it cause the problem?
My upstream's parameters are shown below:
```
function create_upstream {
upstream_id=$1
req_body=$2
#jy: 去除请求体中的空格
req_body=`echo ${req_body} | sed 's/ //g'`
#echo ${req_body}
curl http://${apisix_ip}:${apisix_port}/apisix/admin/upstreams/${upstream_id} -H "X-API-KEY: ${api_key}" -X PUT -d ''${req_body}''
}
create_upstream ${upstream_id} '{
"type":"roundrobin",
"nodes":'${node_info}',
"timeout": {
"connect": 15,
"send": 15,
"read": 15
},
"checks": {
"active": {
"http_path": "/",
"healthy": {
"interval": 2,
"successes": 2
},
"unhealthy": {
"interval": 2,
"http_failures": 3
}
}
},
"keepalive_pool.size": 100,
"keepalive_pool.idle_timeout": 361,
"keepalive_pool.requests": 10000,
"name": "'${upstream_name}'",
"desc": "'${upstream_desc}'"
}'
```
I notice that the default timeout is 15, but the error in my log file seems not related to it, I will try to set it bigger and try to see again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] JiaYueHuang edited a comment on issue #6034: bug: problems caused by stream_proxy (tcp proxy)
Posted by GitBox <gi...@apache.org>.
JiaYueHuang edited a comment on issue #6034:
URL: https://github.com/apache/apisix/issues/6034#issuecomment-1008485771
I use apisix to encapsulate the openstack’s API to achieve high availability and rate limiting.
### Problem description
My apisix configuration file (config.yaml) shows below (only shows the problem related part):
I use the stream_proxy (tcp proxy) to listen ``vip`` (provided by keepalived) and 11211, 3306 ports (related to memcached and mysql services), and then I will create ``upstream`` and ``route`` related to memcached and mysql services to achive High Availability (shown below).
Notice: I observed that, for msyql and memcache services, stream_proxy (tcp proxy) must be used, otherwise (if still use the http proxy), mysql and memcache services wouldn’t work out (commands below couldn’t connect correctly).
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
For simplicity, related upstreams has only one node:
Mysql’s upstream shows blow:
Memcached’s upstream shows below:
Mysql’s stream_route (experiment shows that it’s also valid for ordinary route, but will have the same problem) shows below (matched by 3306 port and forward the service to specific node):
Memcache’s stream_route :
Then we test the mysql and memcache services, execute the command below (it shows correctly):
`telnet <vip> 11211`
`mysql -ukeystone -pKeyStone_pwd -h<vip>`
### 1) Problems caused by nova-api (memcached related)
Configuration related to memcached in nova.conf (if use the ``vip``, there are some problems in nova-api.log, if use specific ip, no ERROR will occur):
### 2) Problems caused by masakari (mysql related)
I alse use the ``vip`` to connect mysql in masakari configuration file (if use real ip, no ERROR will occur):
### Summary
Because I only set one node in upstream, and evidence shows that apisix is functioning correctly, but it still cause some minor problem, which seems related to stream_proxy (tcp proxy), and belows are my question about the apisix’s stream_proxy:
1.Does the stream_proxy (tcp proxy) is a short connection? And if so, it will interrupt in the client side, which will cause the server’s side to trigger some error log (refer to: https://www.jb51.net/article/57602.htm).
2.Is there any alternative solution to connect mysql and memcached with apisix?
If I use the haproxy to achieve High Availability, problems above will not occur (but haproxy doesn’t have rate limiting functionality).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org