You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "fcabbage (via GitHub)" <gi...@apache.org> on 2025/12/18 08:38:13 UTC
[I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
fcabbage opened a new issue, #15880:
URL: https://github.com/apache/dubbo/issues/15880
### Pre-check
- [x] I am sure that all the content I provide is in English.
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.
### Apache Dubbo Component
Java SDK (apache/dubbo)
### Dubbo Version
Dubbo Java 3.3.0, OpenJDK 1.8, Springboot 2.7.18
### Steps to reproduce this issue
One consumer instance and one provider instance and nacos server, just shutdown provider instance.
And the consumer keep executing the reconnection, after all providers are shutdown.
### What you expected to happen
Logs：
```
2025-12-18 16:22:03.563 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:03.593 INFO 39120 --- [andler-thread-1] o.a.d.r.e.s.h.HeaderExchangeHandler : [DUBBO] ChannelReadOnly set true for channel: NettyChannel [channel=[id: 0xb52f26a2, L:/172.16.14.122:53220 - R:/172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Received instance notification, serviceName: MSEProvider, instances: 0, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] 0 unique working revisions: , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.188 INFO 39120 --- [ncesChangeEvent] .r.c.e.l.ServiceInstancesChangedListener : [DUBBO] Notify service com.cabbage.demo.mse.common.api.UserService:default:null with urls 1, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.191 WARN 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received url with EMPTY protocol from registry ServiceDiscoveryRegistryDirectory(registry: 127.0.0.1:8848, subscribed key: [MSEProvider, MSEServiceProvider, Provider])-Directory(invokers: 1[172.16.14.122:18616], validInvokers: 1[172.16.14.122:18616], invokersToReconnect: 0[]), will clear all available addresses., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 4-1. This may be caused by , go to https://dubbo.apache.org/faq/4/1 to find instructions.
2025-12-18 16:22:04.193 INFO 39120 --- [ncesChangeEvent] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0x6733d82a, L:/172.16.14.122:53239 - R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .r.c.m.DefaultMigrationAddressComparator : [DUBBO] No instance address available, stop compare., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:04.197 INFO 39120 --- [ncesChangeEvent] .d.r.c.ServiceDiscoveryRegistryDirectory : [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.cabbage.demo.mse.common.api.UserService:default. Urls Size : 1. Invokers Size : 0. Available Size: 0. Available Invokers : empty, dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.203 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] **The client has been closed and will not reconnect**. , dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:07.204 INFO 39120 --- [lientWorker-5-1] o.a.d.r.t.netty4.NettyConnectionHandler : [DUBBO] The connection [id: 0x6733d82a, L:/172.16.14.122:53239 ! R:/172.16.14.122:18616] of 172.16.14.122:53239 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:13.527 INFO 39120 --- [lientWorker-4-1] o.a.d.r.t.netty4.NettyClientHandler : [DUBBO] The connection [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616] of 172.16.14.122:53220 -> 172.16.14.122:18616 is disconnected., dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.839 INFO 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:28.840 INFO 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyChannel : [DUBBO] Close netty channel [id: 0xb52f26a2, L:/172.16.14.122:53220 ! R:/172.16.14.122:18616], dubbo version: 3.3.0, current host: 172.16.14.122
2025-12-18 16:22:30.892 ERROR 39120 --- [onnect-thread-1] o.a.d.r.transport.netty4.NettyClient : [DUBBO] Failed to connect to provider server by other reason., dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-1. This may be caused by network disconnected, go to https://dubbo.apache.org/faq/6/1 to find instructions.
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /172.16.14.122:18616
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_472]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[na:1.8.0_472]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
at java.lang.Thread.run(Thread.java:750) [na:1.8.0_472]
2025-12-18 16:22:30.894 ERROR 39120 --- [onnect-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Fail to connect toHeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.16.14.122:53220 -> /172.16.14.122:18616]], dubbo version: 3.3.0, current host: 172.16.14.122, error code: 6-16. This may be caused by , go to https://dubbo.apache.org/faq/6/16 to find instructions.
```
The log tells that **the client has been closed and will not reconnect**, but the consumer keep executing the reconnection.
In the K8s environment, since the provider pod will have a new IP after restart, the consumer keeps reporting errors of attempting to connect to the old IP, which continuously triggers alerts.
### Anything else
_No response_
### Are you willing to submit a pull request to fix on your own?
- [ ] Yes I am willing to submit a pull request on my own!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "fcabbage (via GitHub)" <gi...@apache.org>.
fcabbage commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3673224446
And I found that if I do not specify a protocol in @DubboReference, the **DubboProtocol** and **TripleProtocol** will both be instantiated. But when i shutdown the provider, only **TripleInvoker** triggers **destroy()**, the shutdown of Reconnection task will not be triggered in the **DubboInvoker.destroy()**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "fcabbage (via GitHub)" <gi...@apache.org>.
fcabbage commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3674412285
### Common
```
public interface UserService {
User getUserById(Long id);
}
@Data
public class User implements Serializable {
private Long id;
private String name;
}
```
### Provider
```
@DubboService(registry = "nacos", version = "default")
public class UserServiceImpl implements UserService {
@Override
public User getUserById(Long id) {
System.out.println(RpcContext.getServerContext().getUrl().getProtocol());
User user = new User();
user.setId(id);
user.setName(String.valueOf(id));
return user;
}
}
```
application.properties
```
dubbo.application.name=MyConsumer
dubbo.application.qosPort=22233
dubbo.registries.nacos.address=nacos://127.0.0.1:8848
dubbo.protocol.triple.max-frame-size=16777215
```
### Consumer
```
@RestController
@RequestMapping("/user")
public class UserController {
@DubboReference(registry = "nacos", version = "default", retries = 0, timeout = 5000)
private UserService userService;
@GetMapping("/{id}")
public User getUserById(@PathVariable Long id) {
return userService.getUserById(id);
}
}
```
application.properties
```
dubbo.registries.nacos.address=nacos://127.0.0.1:8848
dubbo.registries.nacos.register-mode=instance
dubbo.protocol.triple.max-frame-size=16777215
dubbo.protocol.name=dubbo
dubbo.protocol.port=20880
dubbo.protocol.ext-protocol=tri
dubbo.protocol.preferred-protocol=tri
# New Infos:
# When I enable the following configuration, the protocol is tri;
# When disabled, it defaults to dubbo;
dubbo.protocol.payload=33554432
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3673728158
This seems to be a bug. The puServer is used for single-port multi-protocol communication, and when handling the peer's active disconnection, it invokes org.apache.dubbo.remoting.api.connection.ConnectionHandler#onGoAway. Currently, only the Triple protocol has implemented this interface, while the Dubbo protocol does not handle connection closure within this interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "fcabbage (via GitHub)" <gi...@apache.org>.
fcabbage commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3669413780
And the issue occurs under the following conditions:
nacos instance-level registration
single port for dual protocols (primary protocol: **dubbo**, extension protocol and preferred protocol: **tri**).
When consumers do not specify a protocol in @DubboReference, two channels are established (one for **tri** protocol and one for **dubbo** protocol), and **the dubbo protocol channel** triggering infinite retries.
The issue is resolved when the configuration is modified to either interface-level registration or dual ports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "fcabbage (via GitHub)" <gi...@apache.org>.
fcabbage commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3695313211
Sorry for not seeing your message in a timely manner.
Based on the above demo, I tested the 3.3.6 version. When the provider goes offline, the consumer no longer reports reconnection errors. **But**, the configuration **dubbo.protocol.preferred-protocol=tri** seems to have been invalidated — there is only one channel using the Dubbo protocol between the consumer and the provider. In this case, when the provider goes offline, the destroy method of the consumer's Dubbo invoker will shut down the reconnection task, hence no error is reported.
In the version 3.3.0, this problem occurred because there were two channels between the consumer and the provider, and all calls were made only through the **Triple invoker**.
Could you tell me how to prioritize the use of Triple when the consumer does not specify a protocol in version 3.3.6?
P.S. The reason I do not specify a protocol in the consumer is that I am in the transition period of upgrading from Dubbo 2 to Dubbo 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "ramangeesingh (via GitHub)" <gi...@apache.org>.
ramangeesingh commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3676576737
Hi, I went through the issue and the recent discussion.
From what I understand, when no protocol is specified in @DubboReference, both Dubbo and Triple protocols are initialized, and on provider shutdown only the TripleInvoker properly destroys and stops reconnection, while the DubboInvoker continues trying to reconnect to the old address.
I’d like to help with this issue. I can try to reproduce it using the provided setup and share my findings, or help with documentation/usage clarification if that’s preferred. Please let me know how I can best contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3695319893
Based on my tests, the consumer uses the Triple protocol by default under the configuration you provided. Why would you think it's the Dubbo protocol?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3688556119
I was unable to reproduce your issue with the latest version. Could you provide a complete demo?
* client
<img width="1486" height="863" alt="Image" src="https://github.com/user-attachments/assets/0414e244-d3e0-4736-b01e-ae8297cf3a41" />
* server
<img width="1492" height="834" alt="Image" src="https://github.com/user-attachments/assets/10d14d02-9863-4ac0-b533-954ce514cfd9" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3674020572
Single-port multi-protocol should refer to the port on the server side, rather than a scenario where the same consumer establishes connections using two protocols. It may be necessary to adjust the default logic so that only the preferred protocol is referenced when no protocol is configured on the consumer side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3688601527
It seems that the infinite reconnection issue occurs in version 3.3.0, so you can try upgrading to the latest version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "EarthChen (via GitHub)" <gi...@apache.org>.
EarthChen commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3674040715
Of course, using two protocols for the same interface within a single application should not result in an error either. Could you provide a minimal reproducible demo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] [Bug] The consumer keep executing the reconnection, after all providers are offline [dubbo]
Posted by "fcabbage (via GitHub)" <gi...@apache.org>.
fcabbage commented on issue #15880:
URL: https://github.com/apache/dubbo/issues/15880#issuecomment-3673251219
Could the official Dubbo team provide some clear usage recommendations when using the **single-port and dual-protocol** feature?
Like, should the protocol be specified in @DubboReference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org