You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "hebrd (via GitHub)" <gi...@apache.org> on 2023/03/29 00:57:31 UTC
[GitHub] [dubbo] hebrd opened a new issue, #11231: 服务IP错乱
hebrd opened a new issue, #11231:
URL: https://github.com/apache/dubbo/issues/11231
<!-- If you need to report a security issue please visit https://github.com/apache/dubbo/security/policy -->
- [x] I have searched the [issues](https://github.com/apache/dubbo/issues) of this repository and believe that this is not a duplicate.
### Environment
* Dubbo version: 3.1.3
* Operating System version: open jdk docker image
* Java version: 1.8
* 注册中心：zookeeper三节点集群
* 推空保护： 已关闭
### Steps to reproduce this issue
偶然性出现服务的IP地址错乱，如A服务的ip为IP1，B服务的ip为IP2，客户端调用A服务时，会跑到B服务的IP2去，且无论如何重启A服务，客户端的IP地址一直指向IP2，除非重启客户端，服务IP才能重新恢复
目前还没办法复现，但是在k8s集群中已经出现数次
If there is an exception, please attach the exception trace:
```
java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #11231: 服务IP错乱
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1488061714
> > > 经过进一步定位，发现在K8S如下情况必然出现该问题：
> > >
> > > * K8S CronJob调度执行完成，但是对应的Pod还未删除，假设此时未被删除的Pod IP未ip1
> > > 
> > > * Provider启动，并且K8S将上一步的ip1分配给了服务提供方的Pod
> > > * 此时消费者调用Provider服务，则必然出现如下错误`java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected.`
> > > * 删除调度任务的Pod之后，服务恢复正常
> > >
> > > @AlbumenJ 这种情况有没有什么处理建议？
> >
> >
> > 这个场景我没懂的是这个 ip1 是还被 cornjob 的 container 给持有，但是却已经被分配给了新的 container？
>
> 在K8S里，cronjob的container处于Completed状态但是未被删除时，ip就可能被分配给新的Pod
那 Dubbo 调用应该也是调用到新的 provider，这个新的 provider 处理不了这个请求？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd commented on issue #11231: 服务IP错乱
Posted by "hebrd (via GitHub)" <gi...@apache.org>.
hebrd commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1415339185
关闭推空保护之后，暂未发现改问题继续出现。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd commented on issue #11231: 服务IP错乱
Posted by GitBox <gi...@apache.org>.
hebrd commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1373051824
谢谢回复。由于整个JVM涉及比较多商业信息，不是很方便全部公开出来。能否指导一下需要哪些关键信息，我看看能否脱敏后贴一些上来。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd closed issue #11231: 服务IP错乱
Posted by "hebrd (via GitHub)" <gi...@apache.org>.
hebrd closed issue #11231: 服务IP错乱
URL: https://github.com/apache/dubbo/issues/11231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #11231: 服务IP错乱
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1373183715
> 谢谢回复。由于整个JVM涉及比较多商业信息，不是很方便全部公开出来。能否指导一下需要哪些关键信息，我看看能否脱敏后贴一些上来。
看一下 org.apache.dubbo.registry.client.ServiceDiscoveryRegistryDirectory#urlInvokerMap 这里面的数据，以及 org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery#curatorFramework 内的原始内容是不是对应的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] 服务IP错乱 [dubbo]
Posted by "CrazyHZM (via GitHub)" <gi...@apache.org>.
CrazyHZM closed issue #11231: 服务IP错乱
URL: https://github.com/apache/dubbo/issues/11231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #11231: 服务IP错乱
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1487854917
这个 "java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected." 的异常是哪个 pod 报的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #11231: 服务IP错乱
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1372157556
在出现问题的时候可以 dump 一份 consumer 内存吗，`jmap -dump:format=b,file=dump.bin xxx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd commented on issue #11231: 服务IP错乱
Posted by "hebrd (via GitHub)" <gi...@apache.org>.
hebrd commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1487797416
经过进一步定位，发现在K8S如下情况必然出现该问题：
* K8S cronjob调度执行完成，但是对应的Pod还未删除，假设此时未被删除的Pod IP未ip1
* Provider启动，并且K8S将上一步的ip1分配给了服务提供方的Pod
* 此时消费者调用Provider服务，则必然出现如下错误`java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected.`
* 删除调度任务的Pod之后，服务恢复正常
@AlbumenJ 这种情况有没有什么处理建议？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd commented on issue #11231: 服务IP错乱
Posted by "hebrd (via GitHub)" <gi...@apache.org>.
hebrd commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1487878670
> > 经过进一步定位，发现在K8S如下情况必然出现该问题：
> >
> > * K8S CronJob调度执行完成，但是对应的Pod还未删除，假设此时未被删除的Pod IP未ip1
> > 
> > * Provider启动，并且K8S将上一步的ip1分配给了服务提供方的Pod
> > * 此时消费者调用Provider服务，则必然出现如下错误`java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected.`
> > * 删除调度任务的Pod之后，服务恢复正常
> >
> > @AlbumenJ 这种情况有没有什么处理建议？
>
> 这个场景我没懂的是这个 ip1 是还被 cornjob 的 container 给持有，但是却已经被分配给了新的 container？
在K8S里，cronjob的container处于Completed状态但是未被删除时，ip就可能被分配给新的Pod
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #11231: 服务IP错乱
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1487864313
> 经过进一步定位，发现在K8S如下情况必然出现该问题：
>
> * K8S CronJob调度执行完成，但是对应的Pod还未删除，假设此时未被删除的Pod IP未ip1
> 
> * Provider启动，并且K8S将上一步的ip1分配给了服务提供方的Pod
> * 此时消费者调用Provider服务，则必然出现如下错误`java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected.`
> * 删除调度任务的Pod之后，服务恢复正常
>
> @AlbumenJ 这种情况有没有什么处理建议？
这个场景我没懂的是这个 ip1 是还被 cornjob 的 container 给持有，但是却已经被分配给了新的 container？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] hebrd commented on issue #11231: 服务IP错乱
Posted by "hebrd (via GitHub)" <gi...@apache.org>.
hebrd commented on issue #11231:
URL: https://github.com/apache/dubbo/issues/11231#issuecomment-1487859996
> 这个 "java.io.IOException: Service com.XXXService with version 0.0.0 not found, invocation rejected." 的异常是哪个 pod 报的
应该是消费者报错：
```Failed to invoke the method methodXXX in the service serviceYYY. Tried 1 times of the providers [10.244.152.79:20880] (1/1) from the registry zookeeper:2181 on the consumer 10.244.232.20 using the dubbo version app. Last error is: Failed to invoke remote method: methodXXX, provider: DefaultServiceInstance{serviceName='sssservice', host='10.244.152.79', port=20880, enabled=true, healthy=true```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org