You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2022/11/04 09:08:51 UTC
[GitHub] [servicecomb-java-chassis] edgar1170 opened a new issue, #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
edgar1170 opened a new issue, #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454
对某个循环调用同一个服务器端的客户端接口进行多次调用[不算很频繁]，wait response会不断增长，直到30s后超时，然后会慢慢发生熔断
陆续出现code=408;msg=CommonExceptionData [message=Request Timeout.]
接着出现InvocationException: InvocationException: code=500;msg=CommonExceptionData [message=No available address found.]
查看调用链之后发现，耗时在wait response上面。
请问是因为从从1.3升级到2.7.5从而某些参数变更出现的问题吗？
客户端与服务器的配置如下：
servicecomb:
accesslog:
enabled: true
pattern: "%t %p %m %H %U %s %b %D"
server:
http2:
useAlpnEnabled: false
rest:
client:
http2:
enabled: false
connection:
maxPoolSize: 50
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1305464471
仔细检查客户端的日志中会出现大量以上The executor's work queue is full的报错
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1308174888
@liubao68 服务器这边发现问题了，升级版本之后发生了线程BLOCK，请问这个为什么会在升级之后出现呀: locked org.apache.servicecomb.provider.pojo.PojoConsumerMetaRefresher@4196491c <---- but blocks 5 other threads!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1303152693
1.3旧版本的时候没有上述的配置，全都用的默认值
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1307033743
@liubao68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1305437815
> 看一下服务端access.log，看看对应请求服务端处理的时间实际花费多久？%D对应的值
服务器处理时间可以看到是非常短的，都花在了get connection上面了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1308270356
或许需要在哪里控制线程总数量？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] fanjiwang1992 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
fanjiwang1992 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1305122991
看一下服务端access.log，看看对应请求服务端处理的时间实际花费多久？%D对应的值
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1307056063
这个日志是 A -> B， A超时， B的日志吧？ 可以抓一下堆栈看下， B在忙啥。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1303161855
你的问题看起来和配置没什么关系。 还是需要分析下耗时的原因或者环节。 建议打开metrics并结合服务端、客户端的情况，分析下可能的环节与问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1307034944
@liubao68 使用测试工具跑测试用例的时候，起初能够看到大量如下报错， 2022-11-07 11:38:37.565 ERROR 2268 --- [http-nio-10.0.3.113-8080-Poller] .c.b.h.TomcatConfig$TomcatThreadExecutor : 156 tomcat Work queue full
2022-11-07 11:38:37.566 WARN 2268 --- [http-nio-10.0.3.113-8080-Poller] org.apache.tomcat.util.net.NioEndpoint : 175 Executor rejected socket [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@ceeaf51:org.apache.tomcat.util.net.NioChannel@1993f1bc:java.nio.channels.SocketChannel[connected local=/10.0.3.113:8080 remote=/10.0.3.1:25788]] for processing
java.util.concurrent.RejectedExecutionException: The executor's work queue is full
at org.apache.catalina.core.StandardThreadExecutor.execute(StandardThreadExecutor.java:178)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1169)
at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:856)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:823)
at java.lang.Thread.run(Thread.java:748)
2022-11-07 11:38:37.566 ERROR 2268 --- [http-nio-10.0.3.113-8080-Poller] .c.b.h.TomcatConfig$TomcatThreadExecutor : 156 tomcat Work queue full
2022-11-07 11:38:37.567 WARN 2268 --- [http-nio-10.0.3.113-8080-Poller] org.apache.tomcat.util.net.NioEndpoint : 175 Executor rejected socket [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4d13e92a:org.apache.tomcat.util.net.NioChannel@43aa2179:java.nio.channels.SocketChannel[connected local=/10.0.3.113:8080 remote=/10.0.3.1:18134]] for processing
java.util.concurrent.RejectedExecutionException: The executor's work queue is full
at org.apache.catalina.core.StandardThreadExecutor.execute(StandardThreadExecutor.java:178)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1169)
at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:856)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:823)
at java.lang.Thread.run(Thread.java:748)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1308209106
线程阻塞的数量会不断增加最后爆掉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1305463854
2022-11-07 11:38:35.067 WARN 2268 --- [http-nio-10.0.3.113-8080-Poller] org.apache.tomcat.util.net.NioEndpoint : 175 Executor rejected socket [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@539c57b0:org.apache.tomcat.util.net.NioChannel@43aa2179:java.nio.channels.SocketChannel[connected local=/10.0.3.113:8080 remote=/10.0.3.190:40858]] for processing
java.util.concurrent.RejectedExecutionException: The executor's work queue is full
at org.apache.catalina.core.StandardThreadExecutor.execute(StandardThreadExecutor.java:178)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1169)
at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:856)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:823)
at java.lang.Thread.run(Thread.java:748)
2022-11-07 11:38:35.068 ERROR 2268 --- [http-nio-10.0.3.113-8080-Poller] .c.b.h.TomcatConfig$TomcatThreadExecutor : 156 tomcat Work queue full
2022-11-07 11:38:35.068 WARN 2268 --- [http-nio-10.0.3.113-8080-Poller] org.apache.tomcat.util.net.NioEndpoint : 175 Executor rejected socket [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@79a5b3d3:org.apache.tomcat.util.net.NioChannel@79f59719:java.nio.channels.SocketChannel[connected local=/10.0.3.113:8080 remote=/10.0.3.190:41792]] for processing
java.util.concurrent.RejectedExecutionException: The executor's work queue is full
at org.apache.catalina.core.StandardThreadExecutor.execute(StandardThreadExecutor.java:178)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1169)
at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:856)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:823)
at java.lang.Thread.run(Thread.java:748)
2022-11-07 11:38:35.069 ERROR 2268 --- [http-nio-10.0.3.113-8080-Poller] .c.b.h.TomcatConfig$TomcatThreadExecutor : 156 tomcat Work queue full
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1308281538
能否提供比较完整的堆栈？你给的信息看不出阻塞在哪个地方了。 或者你可以看下完整的堆栈，看看同步代码正在执行的线程，为什么没有释放锁。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1303182452
TPS上升的时候， wait-response 会增长的。 一般的业务统计耗时，只是某条语句的执行时间，通常波动很小。 耗时的部分都是在排队、调度等。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] edgar1170 commented on issue #3454: 升级ServiceComb，从1.3升级到2.7.5的过程中遇到了一个问题
Posted by GitBox <gi...@apache.org>.
edgar1170 commented on issue #3454:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3454#issuecomment-1303168473
>
好的，不过我用系统查调用链的时候，发现服务端业务耗时并不长，大头的耗时都在等待上了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org