You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "JadeYeung (via GitHub)" <gi...@apache.org> on 2023/04/25 17:23:59 UTC
[GitHub] [dubbo] JadeYeung opened a new issue, #12189: Zookeeper集群短暂宕机后，集群恢复受阻
JadeYeung opened a new issue, #12189:
URL: https://github.com/apache/dubbo/issues/12189
### Environment
* Dubbo version: 2.7.x
* Operating System version: Linux RedHat
* Java version: 1.8
### Steps to reproduce this issue
1. Zookeeper集群宕机过半节点，集群不可用
2. 在Zookeeper每个节点启动前，客户端依旧保持不断重试连接的状态
3. Zookeeper宕机节点启动缓慢，选主十分困难
4. 由于我们的设计是将Zookeeper设计成一级注册中心，注册服务较多。怀疑是客户端连接过多，瞬间过来的重连流量导致了网络阻塞，目前稳定下来后，2181端口大约有1.7kt的tcp监听。
现有的一些ZK的数据：
1、实际服务大约1.8w，ZNode节点约33w
2、监听watcher数900多w
目前想到的方案只有扩容Zookeeper集群，毋庸置疑的，现在现象可以看到压力已经达到瓶颈，集群扩容可以减少瞬时重连流量的击垮风险。想来请教，大家实施过程中有没有遇到这个问题，是否还有其他可行方案？
是否还可以进行优化，客户端重连时添加随机时间，减少瞬时重连流量的概率？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #12189: Zookeeper集群短暂宕机后，集群恢复受阻
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #12189:
URL: https://github.com/apache/dubbo/issues/12189#issuecomment-1522651261
可以升级到 Dubbo 3.x 的应用级服务发现模型，会大幅减少注册中心的负载
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] JadeYeung commented on issue #12189: Zookeeper集群短暂宕机后，集群恢复受阻
Posted by "JadeYeung (via GitHub)" <gi...@apache.org>.
JadeYeung commented on issue #12189:
URL: https://github.com/apache/dubbo/issues/12189#issuecomment-1523387549
> 遇到过一次宕机后起不来的情况场景与解决方案如下
> https://mp.weixin.qq.com/s/ToFiDNZj3QscMBbAIiy8ng
太有用了，完全没往配置文件这里考虑，我们用的是3.5.4版本，虽然从文章上看这个版本应该已经解决了这个问题，但是提供了个新的思路排查这个问题，非常感谢！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] songxiaosheng commented on issue #12189: Zookeeper集群短暂宕机后，集群恢复受阻
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #12189:
URL: https://github.com/apache/dubbo/issues/12189#issuecomment-1523354889
遇到过一次宕机后起不来的情况场景与解决方案如下
https://mp.weixin.qq.com/s/ToFiDNZj3QscMBbAIiy8ng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] songxiaosheng commented on issue #12189: Zookeeper集群短暂宕机后，集群恢复受阻
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #12189:
URL: https://github.com/apache/dubbo/issues/12189#issuecomment-1523398150
> > 可以升级到 Dubbo 3.x 的应用级服务发现模型，会大幅减少注册中心的负载
>
> 多谢回复，确实最近也在调研考虑升级，但是由于老旧应用比较多，推动改造难度比较大，但是正在努力💪
新应用用新的，看应用慢慢升
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] JadeYeung commented on issue #12189: Zookeeper集群短暂宕机后，集群恢复受阻
Posted by "JadeYeung (via GitHub)" <gi...@apache.org>.
JadeYeung commented on issue #12189:
URL: https://github.com/apache/dubbo/issues/12189#issuecomment-1523384184
> 可以升级到 Dubbo 3.x 的应用级服务发现模型，会大幅减少注册中心的负载
多谢回复，确实最近也在调研考虑升级，但是由于老旧应用比较多，推动改造难度比较大，但是正在努力💪
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org