You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/05/04 05:13:09 UTC
[GitHub] [shardingsphere] wsg96321 opened a new issue, #17308: connection leak for sharding proxy
wsg96321 opened a new issue, #17308:
URL: https://github.com/apache/shardingsphere/issues/17308
version: sharding proxy v 5.1.0
I deploy a sharding proxy with v5.1.0 , and there is many python script in other machine. The python script connect sharding proxy use intranet connection and it is short connection . But when the sharding proxy run for a few days (about 1-2 day maybe ) , I found there is many connection with TCP state `CLOSE_WAIT` and it is `TIME_WAIT` in client machine(python script's machine).
I use this shell script in local machine , it also happened sometimes .
```
for i in {0..1000000}
do
startTime=`date +%s`
startDate=`date`
mysql -uroot -p111111 -P8777 -h127.0.0.1 -A -e "select 1" ;
endTime=`date +%s`
endDate=`date`
echo "startDate=>$startDate|endDate=>$endDate, calc time => `expr $endTime - $startTime`"
sleep 3;
done
```
when this script run for some time , it will also block , and then I can't create a new connection for this sharding proxy for any client. I think maybe the reason of it is that there are too many `CLOSE_WAIT` TCP connection for this thread. But sometimes this error just for current connection( if I create a new connection there is no block )
the result of this commond `netstat -nalp | grep 8777 | grep 15648 | wc -l` is just `55~57`
the recent time of this error is `May 3 07:11` and `May 3 07:15`
but I have found no useful message in log file ,just a `WARN` message of it below: （there is no `WARN` message for 07:11）
```
[sharding-proxy] [2022-05-03 07:15:36] [WARN] epollEventLoopGroup-3-39 i.n.c.DefaultChannelPipeline:onUnhandledInboundException:1152 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
```
this is my server config,other config just use DistSQL to config datasource and table rules :
```
executor.size: 32
allow.range.query.with.inline.sharding: false
proxy.transaction.type: LOCAL
sql.show: false
proxy.opentracing.enabled: false
query.with.cipher.column: true
proxy.frontend.flush.threshold: 1
acceptor.size: 32
max.connections.size.per.query: 1
proxy.hint.enabled: true
```
PS: There is no mistake if I restart the sharding proxy program. But it will happen just some days after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #17308:
URL: https://github.com/apache/shardingsphere/issues/17308#issuecomment-1118361059
Hi @wsg96321
Your server config is not the 5.x style. You may refer to:
https://github.com/apache/shardingsphere/blob/1699736d88db8c388413229d179eeae3da705a8b/shardingsphere-proxy/shardingsphere-proxy-bootstrap/src/main/resources/conf/server.yaml#L64-L85
The WARN log `An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.` usually doesn't matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN closed issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
TeslaCN closed issue #17308: connection leak for sharding proxy
URL: https://github.com/apache/shardingsphere/issues/17308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #17308:
URL: https://github.com/apache/shardingsphere/issues/17308#issuecomment-1199086883
Hi @wsg96321
We cannot reproduce this issue.
Some connection leak issues are fixed, such as:
- https://github.com/apache/shardingsphere/issues/18634
- https://github.com/apache/shardingsphere/issues/18734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #17308:
URL: https://github.com/apache/shardingsphere/issues/17308#issuecomment-1120140542
I tried the following commands and created about 3000 `TIME_WAIT` in my environments. But the Proxy still worked well.
```
for i in {0..1000000}
do
mysql -uroot -p111111 -P8777 -h127.0.0.1 -A -e "select 1"
done
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] TeslaCN commented on issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
TeslaCN commented on issue #17308:
URL: https://github.com/apache/shardingsphere/issues/17308#issuecomment-1118373093
I tried the script you provided but failed to reproduce. We need you to help us reproduce this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] wsg96321 commented on issue #17308: connection leak for sharding proxy
Posted by GitBox <gi...@apache.org>.
wsg96321 commented on issue #17308:
URL: https://github.com/apache/shardingsphere/issues/17308#issuecomment-1120108800
> Hi @wsg96321 Your server config is not the 5.x style. Could you `cat server.yaml`? You may refer to the following links for new props format:
>
> https://github.com/apache/shardingsphere/blob/1699736d88db8c388413229d179eeae3da705a8b/shardingsphere-proxy/shardingsphere-proxy-bootstrap/src/main/resources/conf/server.yaml#L64-L85
>
> The WARN log `An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.` usually doesn't matter.
Hi, I konw this config different, but thanks alot . The reason that I show this config because I just `cat` the file of the sharding proxy which running with `File mode` , then it flush config in file. Maybe it is right.
The function of this script is just connect a sharding proxy with port(8777,default is 3307) by `mysql client` without interactive for 3 seconds .It doesn't matter for this script , just for a test . The important thing is that it produce too much `TIME_WAIT` connection , and when it's count reach 30~50, the connection will block and can't get a new connection. Is there any better suggestion or direction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org