You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2022/08/19 06:22:53 UTC
[GitHub] [dubbo] wenlincheng opened a new issue, #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
wenlincheng opened a new issue, #10484:
URL: https://github.com/apache/dubbo/issues/10484
* Dubbo version: 3.0.10
* Java version: 1.8
* SpringBoot：2.3.2
* Nacos: 2.0.3
### provider 重启中打印的日志，可以看到绑定的ip为 172.30.19.7
```
[DUBBO] Ignore empty notify urls for subscribe url provider://172.30.19.7:20880/com.xxxxxx.service.SubMailService?anyhost=true&application=ark-sailor&background=false&bind.ip=172.30.19.7&bind.port=20880&category=configurators&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=com.daddylab.ark.sailor.open.service.SubMailService&logger=slf4j&methods=verifyCode,send
```
### provider 重启后 consumer 打印日志 172.30.19.10:48052 -> /172.30.19.36:20880 is disconnected 连接已经断开
```
2022-08-19 14:02:28.031 INFO 7 --- [NettyClientWorker-12-2] o.a.d.r.t.netty4.NettyClientHandler : sampled: [DUBBO] The connection of /172.30.19.10:42880 -> /172.30.19.36:20880 is disconnected., dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:19.028 INFO 7 --- [client-idleCheck-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : sampled: [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [/172.30.19.10:42880 -> /172.30.19.36:20880]], dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:19.028 INFO 7 --- [client-idleCheck-thread-1] o.a.d.r.transport.netty4.NettyChannel : sampled: [DUBBO] Close netty channel [id: 0xb0d101c7, L:/172.30.19.10:42880 ! R:/172.30.19.36:20880], dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:19.030 INFO 7 --- [client-idleCheck-thread-1] o.a.d.r.transport.netty4.NettyClient : sampled: [DUBBO] Close old netty channel [id: 0xb0d101c7, L:/172.30.19.10:42880 ! R:/172.30.19.36:20880] on create new netty channel [id: 0xaf4964fc, L:/172.30.19.10:48052 - R:/172.30.19.36:20880], dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:19.030 INFO 7 --- [NettyClientWorker-12-3] o.a.d.r.t.netty4.NettyClientHandler : sampled: [DUBBO] The connection of /172.30.19.10:48052 -> /172.30.19.36:20880 is established., dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:19.030 INFO 7 --- [client-idleCheck-thread-1] o.a.d.remoting.transport.AbstractClient : sampled: [DUBBO] Successfully connect to server /172.30.19.36:20880 from NettyClient 172.30.19.10 using dubbo version 3.0.10, channel is NettyChannel [channel=[id: 0xaf4964fc, L:/172.30.19.10:48052 - R:/172.30.19.36:20880]], dubbo version: 3.0.10, current host: 172.30.19.10
2022-08-19 14:03:29.031 INFO 7 --- [NettyClientWorker-12-3] o.a.d.r.t.netty4.NettyClientHandler : sampled: [DUBBO] The connection of /172.30.19.10:48052 -> /172.30.19.36:20880 is disconnected.
```
### 调用报错，consumer仍然找provider旧的ip地址172.30.19.36 ，provider重启后ip地址已经变为了 172.30.19.7
```
Failed to invoke the method getOptUser in the service com.xxxxx.system.service.ISysUserService.
Tried 3 times of the providers [172.30.19.36:20880] (1/1) from the registry 172.16.12.151:8848 on the consumer 172.30.19.10 using the dubbo version 3.0.10.
Last error is: Invoke remote method timeout. method: getOptUser,
provider: dubbo://172.30.19.36:20880/com.xxxxxx.system.service.ISysUserService?anyhost=true&application=ark-sailor&background=false&category=providers&check=false&deprecated=false&d
```
### ### 以上情况在本地环境不会出现，项目部署到k8s后，修改代码重新发布后就会出现，手动重启provier的pod后consumer又能订阅到新的ip定制
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1222033143
> 最新的dubbo就是这个样，如果provider的ip地址变了，那么除非重启客户端，否则客户端会一直尝试连接旧地址。 社区反馈的理由是生产环境下服务实例数不能直接变为0，你如果把某个服务的实例变0，最后那个实例会变成不死鸟，所有客户端都会不间断地去尝试重连，即使后来再启动了1个服务实例也如此，直到你受不了把所有客户端都重启了事。 避免这个问题，每个dubbo服务至少要部署2个实例，还不能同时关闭或重启，时刻要留1个实例。 万一服务端因为停电了都挂了，那你就认栽了吧，dubbo客户端会给你留一个不死鸟的。
这么设计啥目的呢。像dev、test环境一般都是单个实例运行，每次发布还要重启消费者，也太麻烦了吧。我刚把两个服务都增加了一个实例，重新发布还是存在问题，设置的注册模式是双注册，双实例部署，A服务注册2个instance 2个interface ,B服务注册了2个instance 2个interface ,修改代码部署重启，A服务注册2个instance 1个interface ,B服务注册了0个instance 1个interface （重启后的情况不一定这样）,就很不稳定。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
zrlw commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1250151798
> 我自己手动在 k8s 上跑了下即使缩容到0然后再扩容也是可以正常调用的。
缩到0后不要马上扩容，注册中心向consumer推送服务数量为0之后再扩容就会出现这个问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220685423
> > > 这个需要看下是不是注册中心地址没更新
> >
> >
> > 注册中心地址应该是不变的吧
>
> 他的意思是在注册中心看 provider 注册 的地址 是不是 变化了
推代码重新部署后provider会注册不上去，在nacos上找不到，k8s手动重启provider的pod就能重新注册，consumer也就能调通。生产环境用的2.7.14版本不存在这个问题，同样是部署在k8s环境，现在是在对dubbo做升级出现的问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] liufeiyu1002 commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
liufeiyu1002 commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220587622
> > 这个需要看下是不是注册中心地址没更新
>
> 注册中心地址应该是不变的吧
他的意思是在注册中心看 provider 注册 的地址 是不是 变化了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] longguiyun commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by "longguiyun (via GitHub)" <gi...@apache.org>.
longguiyun commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1508285058
遇到过这种情况，查看了k8s pod 的配置，然后降低了dubbo服务配置的线程数。问题就没有出现了。猜测可能是k8s pod的资源配置不太够。导致dubbo服务发现更新提供者数据的线程没有执行。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP [dubbo]
Posted by "peopleone (via GitHub)" <gi...@apache.org>.
peopleone commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1866540408
> > consumer
>
> 我们这边dubbo2.x的最新版本，再k8s provider滚动重启的情况下consumer依然有概率(比较偶发)会一直连老的ip调用服务，导致服务调用失败，而且这个报错会一直持续无法恢复。
>
> 这个问题跟ReconnectTimerTask那个重连老ip还不一样，那个顶多是心跳失败，不影响业务调用，我碰到的问题直接影响到业务调用。
>
> 请问各位该怎么解决啊，生产环境也不可能经常去重启consumer服务。
我们生产近期也出现了类似问题，consumer收到了provider变更的notify，并有close netty channel，但是仍然使用old provider ip去调用服务，报错几次后在无notify url的情况下又恢复正常了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1223672632
> 我自己手动在 k8s 上跑了下即使缩容到0然后再扩容也是可以正常调用的。
好的，多谢解答。目前怀疑k8s环境与nacos（非k8s、3实例集群）之间通信可能有问题。
启动日志分析
发现dev环境启动的时候，nacos-client向发送nacos-server发送健康检查报 Server healthy check fail，会切换到别的节点。
注册中心地址配置： nacos://172.16.12.172:8848?backup=172.16.12.193:8848,172.16.12.151:8848
```
1060 Success to connect to server [172.16.12.151:8848] on start
2022-08-22 17:58:49.099 [main] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0]Register server push request handler:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$$Lambda$305/299395997
2022-08-22 17:58:49.099 [main] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0]Registry connection listener to current client:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$1
2022-08-22 17:58:49.099 [main] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0]RpcClient init, ServerListFactory =com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$2
2022-08-22 17:58:49.100 [main] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0] Try to connect to server on start up, server: {serverIp='172.16.12.151', server main port=8848}
2022-08-22 17:58:49.242 [main] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0] Success to connect to server [172.16.12.151:8848] on start up,connectionId=1661162329114_192.168.4.32_49180
1115 Server healthy check fail [172.16.12.151:8848] -> success to connect a server [172.16.12.172:8848]
2022-08-22 17:58:59.122 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0]Server healthy check fail,currentConnection=1661162329114_192.168.4.32_49180
2022-08-22 17:58:59.122 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0] try to re connect to a new server ,server is not appointed,will choose a random server.
2022-08-22 17:58:59.243 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0] success to connect a server [172.16.12.172:8848],connectionId=1661162339139_192.168.4.32_57996
2022-08-22 17:58:59.243 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [e3b9af18-fa05-43c7-9395-3866e5f3cf23_config-0] Abandon prev connection ,server is 172.16.12.151:8848, connectionId is 1661162329114_192.168.4.32_49180
1202 Success to connect to server [172.16.12.172:8848] on start
2022-08-22 17:59:11.955 [main] INFO com.alibaba.nacos.common.remote.client - [9659fdfb-8a60-47b9-ab9f-d6f29be3240f]Register server push request handler:com.alibaba.nacos.client.naming.remote.gprc.NamingPushRequestHandler
2022-08-22 17:59:11.955 [main] INFO com.alibaba.nacos.common.remote.client - [9659fdfb-8a60-47b9-ab9f-d6f29be3240f] Try to connect to server on start up, server: {serverIp='172.16.12.172', server main port=8848}
2022-08-22 17:59:12.066 [main] INFO com.alibaba.nacos.common.remote.client - [9659fdfb-8a60-47b9-ab9f-d6f29be3240f] Success to connect to server [172.16.12.172:8848] on start up,connectionId=1661162313164_192.168.4.32_55434
2022-08-22 17:59:12.066 [main] INFO com.alibaba.nacos.common.remote.client - [9659fdfb-8a60-47b9-ab9f-d6f29be3240f]Register server push request
1233 Success to connect to server [172.16.12.172:8848] on start
2022-08-22 17:59:12.437 [main] INFO com.alibaba.nacos.common.remote.client - [b5b4dc4a-146d-444c-be71-f74a97f1aa4e_config-0] Try to connect to server on start up, server: {serverIp='172.16.12.172', server main port=8848}
2022-08-22 17:59:12.545 [main] INFO com.alibaba.nacos.common.remote.client - [b5b4dc4a-146d-444c-be71-f74a97f1aa4e_config-0] Success to connect to server [172.16.12.172:8848] on start up,connectionId=1661162313164_192.168.4.32_55434
2022-08-22 17:59:12.546 [main] INFO com.alibaba.nacos.common.remote.client - [b5b4dc4a-146d-444c-be71-f74a97f1aa4e_config-0]Register server push request
1218 Success to connect to server [172.16.12.193:8848] on start
2022-08-22 17:59:12.233 [main] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] Try to connect to server on start up, server: {serverIp='172.16.12.193', server main port=8848}
2022-08-22 17:59:12.364 [main] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] Success to connect to server [172.16.12.193:8848] on start up,connectionId=1661162352239_192.168.4.32_32950
2022-08-22 17:59:12.365 [main] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d]Register server push request
3039 Server healthy check fail [172.16.12.193:8848] -> success to connect a server [172.16.12.151:8848]
2022-08-22 18:00:12.239 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d]Server healthy check fail,currentConnection=1661162352239_192.168.4.32_32950
2022-08-22 18:00:12.239 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] try to re connect to a new server ,server is not appointed,will choose a random server.
2022-08-22 18:00:12.359 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] success to connect a server [172.16.12.151:8848],connectionId=1661162412250_192.168.4.32_57666
2022-08-22 18:00:12.359 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] Abandon prev connection ,server is 172.16.12.193:8848, connectionId is 1661162352239_192.168.4.32_32950
2022-08-22 18:00:12.359 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d]Notify disconnected event to listeners
2022-08-22 18:00:12.359 [com.alibaba.nacos.client.remote.worker] WARN com.alibaba.nacos.client.naming - Grpc connection disconnect, mark to redo
3372 Server healthy check fail [172.16.12.151:8848] -> success to connect a server [172.16.12.172:8848]
2022-08-22 18:00:22.365 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d]Server healthy check fail,currentConnection=1661162412250_192.168.4.32_57666
2022-08-22 18:00:22.365 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] try to re connect to a new server ,server is not appointed,will choose a random server.
2022-08-22 18:00:22.473 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] success to connect a server [172.16.12.172:8848],connectionId=1661162313164_192.168.4.32_55434
2022-08-22 18:00:22.473 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d] Abandon prev connection ,server is 172.16.12.151:8848, connectionId is 1661162412250_192.168.4.32_57666
2022-08-22 18:00:22.474 [com.alibaba.nacos.client.remote.worker] INFO com.alibaba.nacos.common.remote.client - [b3a2ce85-3457-4828-b846-d1fcc6f1425d]Notify disconnected event to listeners
2022-08-22 18:00:22.474 [com.alibaba.nacos.client.remote.worker] WARN com.alibaba.nacos.client.naming - Grpc connection disconnect, mark to redo
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220314678
这个需要看下是不是注册中心地址没更新
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220314698
这个需要看下是不是注册中心地址没更新
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
zrlw commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1221836175
最新的dubbo就是这个样，如果provider的ip地址变了，那么除非重启客户端，否则客户端会一直尝试连接旧地址。
社区反馈的理由是生产环境下服务实例数不能直接变为0，你如果把某个服务的实例变0，最后那个实例会变成不死鸟，所有客户端都会不间断地去尝试重连，即使后来再启动了1个服务实例也如此，直到你受不了把所有客户端都重启了事。
避免这个问题，每个dubbo服务至少要部署2个实例，还不能同时关闭或重启，时刻要留1个实例。
万一服务端因为停电了都挂了，那你就认栽了吧，dubbo客户端会给你留一个不死鸟的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1222084929
我自己手动在 k8s 上跑了下即使缩容到0然后再扩容也是可以正常调用的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] pingod commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
pingod commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1250051382
这个问题最后没见解决，issues就关闭了？？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] liufeiyu1002 commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
liufeiyu1002 commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220713331
>
> > > > 这个需要看下是不是注册中心地址没更新
> > >
> > >
> > > 注册中心地址应该是不变的吧
> >
> >
> > 他的意思是在注册中心看 provider 注册 的地址 是不是 变化了
> > 推代码重新部署后provider会注册不上去，在nacos上找不到，k8s手动重启provider的pod就能重新注册，consumer也就能调通。生产环境用的2.7.14版本不存在这个问题，同样是部署在k8s环境，现在是在对dubbo做升级出现的问题。
最好是能看下 `推代码重新部署provider` 这个过程的日志
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220342253
> 这个需要看下是不是注册中心地址没更新
一直是这个 172.16.12.151:8848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1221762954
> >
>
> > > > > 这个需要看下是不是注册中心地址没更新
> > > >
> > > >
> > > > 注册中心地址应该是不变的吧
> > >
> > >
> > > 他的意思是在注册中心看 provider 注册 的地址 是不是 变化了
> > > 推代码重新部署后provider会注册不上去，在nacos上找不到，k8s手动重启provider的pod就能重新注册，consumer也就能调通。生产环境用的2.7.14版本不存在这个问题，同样是部署在k8s环境，现在是在对dubbo做升级出现的问题。
>
> 最好是能看下 `推代码重新部署provider` 这个过程的日志
在pod里启动会报这个错，本地启动不报
```
2022-08-22 11:36:14.590 [main] DEBUG c.a.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader - com_alibaba_nacos_shaded_io_grpc_netty_shaded_netty_transport_native_epoll cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no com_alibaba_nacos_shaded_io_grpc_netty_shaded_netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.epoll.Native.<clinit>(Native.java:57)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.grpc.netty.Utils.isEpollAvailable(Utils.java:220)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.grpc.netty.Utils.<clinit>(Utils.java:92)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.<clinit>(NettyChannelBuilder.java:72)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:37)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider.builderForAddress(NettyChannelProvider.java:23)
at com.alibaba.nacos.shaded.io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.createNewChannelStub(GrpcClient.java:97)
at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:265)
at com.alibaba.nacos.common.remote.client.RpcClient.start(RpcClient.java:394)
at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.ensureRpcClient(ClientWorker.java:941)
at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.getOneRunningClient(ClientWorker.java:1104)
at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.queryConfig(ClientWorker.java:996)
at com.alibaba.nacos.client.config.impl.ClientWorker.getServerConfig(ClientWorker.java:407)
at com.alibaba.nacos.client.config.NacosConfigService.getConfigInner(NacosConfigService.java:166)
at com.alibaba.nacos.client.config.NacosConfigService.getConfig(NacosConfigService.java:94)
at com.alibaba.cloud.nacos.client.NacosPropertySourceBuilder.loadNacosData(NacosPropertySourceBuilder.java:85)
at com.alibaba.cloud.nacos.client.NacosPropertySourceBuilder.build(NacosPropertySourceBuilder.java:73)
at com.alibaba.cloud.nacos.client.NacosPropertySourceLocator.loadNacosPropertySource(NacosPropertySourceLocator.java:199)
at com.alibaba.cloud.nacos.client.NacosPropertySourceLocator.loadNacosDataIfPresent(NacosPropertySourceLocator.java:186)
at com.alibaba.cloud.nacos.client.NacosPropertySourceLocator.loadNacosConfiguration(NacosPropertySourceLocator.java:158)
at com.alibaba.cloud.nacos.client.NacosPropertySourceLocator.loadSharedConfiguration(NacosPropertySourceLocator.java:116)
at com.alibaba.cloud.nacos.client.NacosPropertySourceLocator.locate(NacosPropertySourceLocator.java:101)
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52)
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47)
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98)
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626)
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at com.daddylab.ark.sailor.admin.ArkSailorApplication.main(ArkSailorApplication.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Suppressed: java.lang.UnsatisfiedLinkError: no com_alibaba_nacos_shaded_io_grpc_netty_shaded_netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at com.alibaba.nacos.shaded.io.grpc.netty.shaded.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 44 common frames omitted
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP [dubbo]
Posted by "elvislou (via GitHub)" <gi...@apache.org>.
elvislou commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1855003511
> consumer
我们这边dubbo2.x的最新版本，再k8s provider滚动重启的情况下consumer依然有概率(比较偶发)会一直连老的ip调用服务，导致服务调用失败，而且这个报错会一直持续无法恢复。
这个问题跟ReconnectTimerTask那个重连老ip还不一样，那个顶多是心跳失败，不影响业务调用，我碰到的问题直接影响到业务调用。
请问各位该怎么解决啊，生产环境也不可能经常去重启consumer服务。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] chickenlj closed issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
chickenlj closed issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
URL: https://github.com/apache/dubbo/issues/10484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220344310
dubbo配置
```
# Dubbo
dubbo:
scan:
# dubbo 服务扫描基准包
base-packages: com.xxxxx.service
# 协议
protocols:
# dubbo 协议
dubbo:
name: dubbo
port: -1
registry:
address: nacos://172.16.12.151:8848/?backup=172.16.12.193:8848,172.16.12.172:8848&username=nacos&password=nacos&namespace=${spring.profiles.active}&group=xxxx
application:
qos-enable: false
logger: slf4j
# interface / instance / all
register-mode: interface
consumer:
check: false
timeout: 5000
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wenlincheng commented on issue #10484: k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP
Posted by GitBox <gi...@apache.org>.
wenlincheng commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1220336317
> 这个需要看下是不是注册中心地址没更新
注册中心地址应该是不变的吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] k8s 环境发布代码，provider重启后consumer无法找到服务提供者，consumer订阅旧的IP [dubbo]
Posted by "zrlw (via GitHub)" <gi...@apache.org>.
zrlw commented on issue #10484:
URL: https://github.com/apache/dubbo/issues/10484#issuecomment-1855204472
dubbo3应该解决这个问题了，但是dubbo2没有相关的机制，所以dubbo3的相关PR没办法用于dubbo2。
具体情况可以问问 @AlbumenJ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org