You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2022/06/15 10:21:59 UTC
[GitHub] [servicecomb-java-chassis] diwenzheng opened a new issue, #3079: 高并发下内存泄露和微服务队列拥塞问题
diwenzheng opened a new issue, #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079
从1.3.0 到现在 2.5.0 版本 ,业务服务原本内存是初始化是1G,MAX 4G,在高tps的冲击下,内存会逐渐增高,直到接近4g,但是如果此时中断业务,内存并不会释放。这会不会框架底层存在内存泄露的情况？
还有一个问题,版本是2.5.0,在接口响应相当慢的时候,频繁冲击业务服务,随后停止业务冲击,发现仍然业务请求打印,持续时间过长,重启服务后恢复.这是否是因为微服务内部队列存在拥塞,即使请求方终止,部分请求会遗留在服务队列中,直至运行完？
@liubao68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] diwenzheng commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
diwenzheng commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1156293153
servicecomb:
service:
environment: development
application: xxx
name: xxx
version: 1.1.0
registry:
address: ${REGISTRY_ADDRESS}
instance:
watch: true
# highway:
# address: 0.0.0.0:7070 # 监听主机的全部网卡IP
rest:
address: 0.0.0.0:${LOCAL_LISNTENING_PORT}
server:
# connection-limit: 1000 # Max allowed client connections
# thread-count: 8 # rest server verticle instance count(Deprecated)
verticle-count: 4 # rest server verticle instance count
compression: false # Wether the server support compression
idleTimeoutInSeconds: 120000 # Timeout for server's idle connection, The idle connections will be closed
requestWaitInPoolTimeout: 1000
client:
# thread-count: 8 # rest server verticle instance count(Deprecated)
verticle-count: 4 # rest server verticle instance count
connection:
maxPoolSize: 100 # The maximum number of connections in each connection pool for an IP:port combination
idleTimeoutInSeconds: 60000 # Timeout for client's idle connection, The idle connections will be closed
# keepAlive: true # Whether to use long connections
compression: false # Wether the client support compression
maxHeaderSize: 8192 # The max header size of the response the client can process, unit is Byte
request:
timeout: 30000
executor:
default:
group: 2
maxThreads-per-group: 50
maxQueueSize-per-group: 500000
handler:
chain:
Provider:
default: qps-flowcontrol-provider,bizkeeper-provider
Consumer:
default: qps-flowcontrol-consumer,bizkeeper-consumer,loadbalance # 降级策略配置 隔离
loadbalance:
retryEnabled: true
retryOnNext: 2
retryOnSame: 1
strategy:
name: RoundRobin # Support RoundRobin,Random,WeightedResponse,SessionStickiness
isolation:
enabled: true
errorThresholdPercentage: 80 #错误率百分80
enableRequestThreshold: 300 #请求数大于5 连续三个周期错误率到达80%
singleTestTime: 60000
continuousFailureThreshold: 3
maxSingleTestWindow: 300000 # 为了保证在并发情况下只有一个实例放通，会锁定放通实例。这个时间表示最大锁定时间。
minIsolationTime: 3000 # 最短隔离时间。并发情况下，实例隔离后进行中的请求可能快速刷新隔离状态，增加最短隔离时间。
recoverImmediatelyWhenSuccess: true
isolation: # 降级策略配置 隔离
Provider:
timeout:
enabled: true
timeoutInMilliseconds: 300000
Consumer:
timeout:
enabled: true
timeoutInMilliseconds: 300000
circuitBreaker:
Provider:
enabled: true
sleepWindowInMilliseconds: 150000
requestVolumeThreshold: 20
Consumer:
enabled: true
sleepWindowInMilliseconds: 150000
requestVolumeThreshold: 20
fallback:
Provider:
enabled: true
Consumer:
enabled: true
fallbackpolicy:
Provider:
policy: throwexception
Consumer:
policy: throwexception
flowcontrol: # 流控
Provider:
qps:
enabled: true
strategy: FixedWindow
global:
limit: 1000 # provider接受请求流量的全局配置 没有具体的流控配置时，此配置生效
Consumer:
qps:
enabled: true
strategy: FixedWindow
global:
limit: 500 # consumer 消耗流量的全局配置
inspector:
enabled: false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1157298967
建议dump一下内容，用内存分析工具先看下泄漏点。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] zhaozw commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
zhaozw commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1157151191
可否查看一下内存快照？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] diwenzheng commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
diwenzheng commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1157168419
对了补充一下,前置服务很多有使用 CseHttpEntity<?> entity = new CseHttpEntity<>(req, null); 异步调用传递参数到该服务。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] diwenzheng commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
diwenzheng commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1156297865
我们是有频繁使用 IncationContext 来传递部分数据的。每次接口都会使用这个,带给相对应的服务的。是否这个存在内存泄露
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] diwenzheng commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
diwenzheng commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1156302643
这是否和这个有些关系？ https://github.com/apache/servicecomb-java-chassis/issues/3025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] diwenzheng commented on issue #3079: 高并发下内存泄露和微服务队列拥塞问题
Posted by GitBox <gi...@apache.org>.
diwenzheng commented on issue #3079:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3079#issuecomment-1157165709
> 可否查看一下内存快照？
这两个都是线上问题,当时处理的时间比较快,所以基本快速恢复了。并没有时间去查看快照。当时现象是4G内存吃满,容器吃满CPU,导致虚拟机CPU占用率抵达100%。我这边仔细看了一下我们业务受理的代码，除了有些异步线程使用JDB1.8特性提供的Commfulture之外,其余对于线程利用率并不高,观察虚拟化平台发现,CPU趋势是逐渐增高至100%,看到#3025这个,我的配置配置比较大,会不会存在因为fullgc,引起的CPU撑爆现象.最近我们也会观察线程cpu 内存使用情况.如果还持续增长,会提供jvm线程日志,内存占用等信息。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org