You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2021/08/02 08:52:26 UTC
[GitHub] [dubbo] gavinWithGu opened a new issue #8399: Zookeeper is no connected when using referenceConfigCache.get
gavinWithGu opened a new issue #8399:
URL: https://github.com/apache/dubbo/issues/8399
- [ ] I have searched the [issues](https://github.com/apache/dubbo/issues) of this repository and believe that this is not a duplicate.
- [ ] I have checked the [FAQ](https://github.com/apache/dubbo/blob/master/FAQ.md) of this repository and believe that this is not a duplicate.
### Environment
* Dubbo version: 2.7.6
* Operating System version: Windows 10
* Java version: 1.8
* Zookeeper Version: 3.6.2
### 业务场景：
本地有一个springboot定时任务，每隔几秒钟触发使用dubbo的泛化调用远程的provider服务：
1. 断点调试：`ReferenceConfigCache.getCache().get()` 会触发CuratorZookeeperClient去连接Zookeeper
2. **CuratorZookeeperClient类：**`boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); `每次返回都为false
3. 但是zookeeper端显示已经连接成功，并且客户端的连接日志也显示的是establishment
**zk端口连接情况：**
**客户端zk的连接日志如下：**
```
2021-08-02 16:17:36.307[org.apache.zookeeper.SaslServerPrincipal][Line:74] [spring-task-SendThread(**.**.**.**:2181)] DEBUG org.apache.zookeeper.SaslServerPrincipal - Canonicalized address to **.**.**.**
2021-08-02 16:17:36.311[org.apache.zookeeper.ClientCnxn$SendThread][Line:1025] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server **.**.**.**/**.**.**.**:2181. Will not attempt to authenticate using SASL (unknown error)
2021-08-02 16:17:36.321[org.apache.zookeeper.ClientCnxn$SendThread][Line:879] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to **.**.**.**/**.**.**.**:2181, initiating session
2021-08-02 16:17:36.324[org.apache.zookeeper.ClientCnxn$SendThread][Line:952] [spring-task-SendThread(**.**.**.**:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on **.**.**.**/**.**.**.**:2181
2021-08-02 16:17:36.338[org.apache.zookeeper.ClientCnxn$SendThread][Line:1299] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server **.**.**.**/**.**.**.**:2181, sessionid = 0x1054f231a03004e, negotiated timeout = 40000
2021-08-02 16:17:36.510[org.apache.zookeeper.ZooKeeper][Line:442] [spring-task-saga1] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=**.**.**.**:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@d234dc3
2021-08-02 16:17:45.530[org.apache.zookeeper.SaslServerPrincipal][Line:74] [spring-task-SendThread(**.**.**.**:2181)] DEBUG org.apache.zookeeper.SaslServerPrincipal - Canonicalized address to **.**.**.**
2021-08-02 16:17:45.530[org.apache.zookeeper.ClientCnxn$SendThread][Line:1025] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server **.**.**.**/**.**.**.**:2181. Will not attempt to authenticate using SASL (unknown error)
2021-08-02 16:17:45.541[org.apache.zookeeper.ClientCnxn$SendThread][Line:879] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to **.**.**.**/**.**.**.**:2181, initiating session
2021-08-02 16:17:45.541[org.apache.zookeeper.ClientCnxn$SendThread][Line:952] [spring-task-SendThread(**.**.**.**:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on **.**.**.**/**.**.**.**:2181
2021-08-02 16:17:45.550[org.apache.zookeeper.ClientCnxn$SendThread][Line:1299] [spring-task-SendThread(**.**.**.**:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server **.**.**.**/**.**.**.**:2181, sessionid = 0x1054f231a03004f, negotiated timeout = 40000
```
### 问题：
1. 为什么client.blockUntilConnected() 每次返回都为false(已经设置timeout，无效果)
2. 每次执行一次`ReferenceConfigCache.getCache().get()` ，我看到zk端口连接都会多一个，这个会持续增长(猜测是因为cache里为null，所以每次获取referenceConfig都会新建连接？)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-904433649
@xiaoheng1 Sorry for the late. I have been busy with other affairs for past few days. Today I checked the source code and find out the reason for this problem is as bellows:
1. There is a State-Machines mechanism in zookeeper client connection
2. When call ` client.start() `, ZK client will start a daemon thread and try to start a client connection to zookeeper server.
3. When call `client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS);` the client will check the state of `currentConnectionState`. if false, the thread will wait until max wait time . Once this happened, **Zookeeper is not connected Exception** will be throw.
4. Meanwhile , once connected, the client will change the `currentConnectionState `and notify the wait thread to return
In my scenario, it seems the daemon thread have to spend longer time than the deafult timeout configuration to get the zk connection.
So I try to increase the timeout paramter and it must to be set in RegistryConfig as bellows:
```
RegistryConfig registryConfig = new RegistryConfig();
...
registryConfig.setTimeout(30000);
...
```
In GenericService call scenario, Yml setting is useless
```
dubbo:
metadata-report:
timeout: 30000
config-center:
timeout: 30000
```
Mark it for myself ^_^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-891557178
> can you provider a demo to reproduce this problem?
Downgrade to dubbo 2.7.3 and everything seems all right ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] xiaoheng1 commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
xiaoheng1 commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-891563734
I can’t get some useful information, if you can provide a demo to reproduce this problem, it will speed up our investigation progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] xiaoheng1 commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
xiaoheng1 commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-890960457
can you provider a demo to reproduce this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu closed issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu closed issue #8399:
URL: https://github.com/apache/dubbo/issues/8399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-891634583
> **Dubbo client :**
>
> ```
> public static GenericService getDubboService(String address, String protocol, String service, String version) {
> ReferenceConfig<GenericService> config = getReference(address, protocol, service, version);
> return ReferenceConfigCache.getCache().get(config);
> }
> ```
>
> **Based on debug result：**
> **ReferenceConfigCache.getCache().get(config )** will call construction method of **CuratorZookeeperClient** as bellow
> `boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); `
> **connected** will always return false and then **Zookeeper is no connected** Exception will be throw
**additional remarks**
Normal dubbo client rpc request goes ok , the problem only occur when I use GenericService invoke. And when downgrade to 2.7.3, the problem disappear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] xiaoheng1 commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
xiaoheng1 commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-894079274
> **Dubbo client :**
>
> ```
> public static GenericService getDubboService(String address, String protocol, String service, String version) {
> ReferenceConfig<GenericService> config = getReference(address, protocol, service, version);
> return ReferenceConfigCache.getCache().get(config);
> }
> ```
>
> **Based on debug result：**
> **ReferenceConfigCache.getCache().get(config )** will call construction method of **CuratorZookeeperClient** as bellow
> `boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); `
> **connected** will always return false and then **Zookeeper is no connected** Exception will be throw
Tested on my local machine and found that connected = false was not found.
system:
macos
jdk1.8
dubbo2.7.6
zk:3.6.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-904447229
The reason why it goes ok on version 2.7.3 is that the implemention of `CuratorZookeeperClient `is different and there is no judgement on `currentConnectionState`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gavinWithGu commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
gavinWithGu commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-891622169
**Dubbo client :**
```
public static GenericService getDubboService(String address, String protocol, String service, String version) {
ReferenceConfig<GenericService> config = getReference(address, protocol, service, version);
return ReferenceConfigCache.getCache().get(config);
}
```
**Based on debug result：**
**ReferenceConfigCache.getCache().get(config )** will call construction method of **CuratorZookeeperClient** as bellow
` boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); `
**connected** will always return false and then **Zookeeper is no connected** Exception will be throw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] xiaoheng1 commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
xiaoheng1 commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-894077380
Can you provide a detailed definition of the getReference method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] xiaoheng1 commented on issue #8399: Zookeeper is no connected when using referenceConfigCache.get
Posted by GitBox <gi...@apache.org>.
xiaoheng1 commented on issue #8399:
URL: https://github.com/apache/dubbo/issues/8399#issuecomment-893982341
@gavinWithGu i will check it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org