You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2022/06/26 12:30:48 UTC

[GitHub] [servicecomb-java-chassis] cuixiushuai opened a new issue, #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

cuixiushuai opened a new issue, #3136:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136

   ### 问题描述
   java chassis客户端访问服务器（以本地注册发现服务的方式），偶现490错误，报错Caused by: io.vertx.core.VertxException: Connection was closed
   2022-06-20 08:04:40.081|[xxx]|transport-vert.x-eventloop-thread-0|ERROR|InvokerUtils.java:333|invoke failed, operation xxx, trace id xx
   org.apache.servicecomb.swagger.invocation.exception.InvocationException: InvocationException: code=490;msg=CommonExceptionData [message=Unexpected consumer error, please check logs for details]
   
   ### 定位过程
   1 分析服务端日志，发现每次出问题时，前边30秒左右，必然有一次有一次正常的请求。由此怀疑是，连接池复用出现问题，30秒为连接被关闭的时间。
   2 根据 https://github.com/apache/servicecomb-java-chassis/issues/2603 分析，可能是服务端参数配置的不合理，服务端提前关闭了客户端的请求，导致客户端准备复用连接时，使用了一个被关闭的连接；也可能是客户端自身出问题，复用了一个被关闭的连接。
   3 进一步通过抓包，定界是客户端问题还是服务端问题。
   ![image](https://user-images.githubusercontent.com/54208931/175813173-0f271a9e-ad73-4b30-b44e-b72de97b5d1e.png)
   抓包发现，出现问题时，确实是客户端复用了一个连接，并且客户端主动关闭了连接，由此定界为客户端问题。
   4 分析代码
   vertx创建一个连接后，会在pipeline上创建一个IdleStateHandler，参考HttpChannelConnector.applyHttp1xConnectionOptions()，启动一个定时任务AllIdleTimeoutTask，定时扫描该连接是否空闲，达到idleTImeout后，关闭该连接。idleTImeout从配置项servicecomb.rest.client.connection.idleTimeoutInSeconds中读取，默认值30秒，与定位过程1中的现象符合。
   由于该定时任务的执行，与从连接池中选连接，没有同步机制保障，导致如果连接池中选取了一个连接，该连接已经30秒未活动，就有可能出现还没来得及使用该连接发送数据，就被AllIdleTimeoutTask关闭的问题。
   5 连接池本身也有空闲检测机制，会超时释放连接
   参考HttpClientImpl构造器中的PoolChecker checker，EXPIRED_CHECKER。由于超时释放与选取连接有同步机制保障，如果由连接池自身的这个机制来保证空连接释放，就不会出现刚选了一个连接，就被关闭的问题。
   由于java-chassis设置连接池自身的超时参数keepAliveTimeout，读取的也是servicecomb.rest.client.connection.idleTimeoutInSeconds，导致空闲连接关闭可能由4种的AllIdleTimeoutTask触发，也可能由本端中的PoolChecker 触发，只要由AllIdleTimeoutTask触发，就可能出问题。
   ### 规避
   定义一个类，继承HttpTransportHttpClientOptionsSPI，在services中定义新类实现spi注册发现，覆写其中的getKeepAliveTimeout，使该配置不再读取servicecomb.rest.client.connection.idleTimeoutInSeconds，设置一个比servicecomb.rest.client.connection.idleTimeoutInSeconds小的值，就能保证，空连接释放是由连接池的PoolChecker 触发。
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3136:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136#issuecomment-1168648994

   使用的是哪个版本？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-java-chassis] cuixiushuai commented on issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

Posted by GitBox <gi...@apache.org>.
cuixiushuai commented on issue #3136:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136#issuecomment-1170070222

   > 使用的是哪个版本？
   2.7.0
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3136:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136#issuecomment-1250983192

   fixed in https://github.com/apache/servicecomb-java-chassis/pull/3346


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-java-chassis] cuixiushuai commented on issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

Posted by GitBox <gi...@apache.org>.
cuixiushuai commented on issue #3136:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136#issuecomment-1170069519

   2.7.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [servicecomb-java-chassis] liubao68 closed issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed

Posted by GitBox <gi...@apache.org>.
liubao68 closed issue #3136: http连接池，客户端复用处于超时边界的连接，导致报错Connection was closed
URL: https://github.com/apache/servicecomb-java-chassis/issues/3136


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org