You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@seata.apache.org by "biedongbin (via GitHub)" <gi...@apache.org> on 2026/02/04 11:52:32 UTC
[I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
biedongbin opened a new issue, #7986:
URL: https://github.com/apache/incubator-seata/issues/7986
### Check Ahead
- [x] I have searched the [issues](https://github.com/seata/seata/issues) of this repository and believe that this is not a duplicate.
- [ ] I am willing to try to fix this bug myself.
### Ⅰ. Issue Description
因seata-server 配置了读写分离的数据库配置 引起了事务问题，因此 修改了配置之后重启sea-server
版本 ：2.5.0
注册中心：nacos
环境：生产环境
部署形式：阿里云K8S集群 docker 内 seata-server.jar
### Ⅱ. Describe what happened
重启之前一切正常，事务正常，链接正常，因读写分离问题，偶发 事务注册之后、再次注册分支事务或者分支事务上报会出现异常， 检查了所有的配置项，未发现有关 心跳检测相关内容。
重启之后 客户端一直在异常， 最后没办法 重启了客户端应用后正常。
`2026-02-04 18:49:58,512 [NettyClientSelector_TMROLE_1_2_4] INFO [] [] [o.a.s.core.rpc.netty.AbstractNettyRemotingClient] - ChannelHandlerContext(AbstractNettyRemotingClient$ClientHandler#0, [id: 0x44dd1c62, L:null ! R:/10.10.240.0:10222]) will closed
2026-02-04 18:49:58,512 [timeoutChecker_1_1] ERROR [] [] [o.a.seata.core.rpc.netty.NettyClientChannelManager] - 0304 register RM failed.
org.apache.seata.common.exception.FrameworkException: can not connect to services-server.
at org.apache.seata.core.rpc.netty.NettyClientBootstrap.getNewChannel(NettyClientBootstrap.java:215)
at org.apache.seata.core.rpc.netty.NettyPoolableFactory.makeObject(NettyPoolableFactory.java:58)
at org.apache.seata.core.rpc.netty.NettyPoolableFactory.makeObject(NettyPoolableFactory.java:34)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doConnect(NettyClientChannelManager.java:309)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.acquireChannel(NettyClientChannelManager.java:113)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:243)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:222)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:174)
at org.apache.seata.core.rpc.netty.AbstractNettyRemotingClient.lambda$init$0(AbstractNettyRemotingClient.java:122)
at com.alibaba.ttl.TtlRunnable.run(TtlRunnable.java:60)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.seata.common.exception.FrameworkException: connect failed, can not connect to services-server.
at org.apache.seata.core.rpc.netty.NettyClientBootstrap.getNewChannel(NettyClientBootstrap.java:188)
... 17 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: Host is unreachable: /10.10.240.0:10222
Caused by: java.net.NoRouteToHostException: Host is unreachable
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-02-04 18:49:58,512 [timeoutChecker_1_1] INFO [] [] [o.a.seata.core.rpc.netty.NettyClientChannelManager] - will connect to 10.10.168.230:10222
2026-02-04 18:49:58,512 [timeoutChecker_1_1] INFO [] [] [o.apache.seata.core.rpc.netty.NettyPoolableFactory] - NettyPool create channel to transactionRole:TMROLE,address:10.10.168.230:10222,msg:< RegisterTMRequest{version='2.5.0', applicationId='salus-distribution-app', transactionServiceGroup='default_tx_group', extraData='ak=null
digest=default_tx_group,10.10.90.50,1770202198512
timestamp=1770202198512
authVersion=V4
vgroup=default_tx_group
ip=10.10.90.50
'} >
2026-02-04 18:49:58,517 [NettyClientSelector_TMROLE_1_3_4] INFO [] [] [o.a.s.core.rpc.netty.AbstractNettyRemotingClient] - ChannelHandlerContext(AbstractNettyRemotingClient$ClientHandler#0, [id: 0x2a0218ca, L:null ! R:/10.10.168.230:10222]) will closed
2026-02-04 18:49:58,517 [timeoutChecker_1_1] ERROR [] [] [o.a.seata.core.rpc.netty.NettyClientChannelManager] - 0304 register RM failed.
org.apache.seata.common.exception.FrameworkException: can not connect to services-server.
at org.apache.seata.core.rpc.netty.NettyClientBootstrap.getNewChannel(NettyClientBootstrap.java:215)
at org.apache.seata.core.rpc.netty.NettyPoolableFactory.makeObject(NettyPoolableFactory.java:58)
at org.apache.seata.core.rpc.netty.NettyPoolableFactory.makeObject(NettyPoolableFactory.java:34)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doConnect(NettyClientChannelManager.java:309)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.acquireChannel(NettyClientChannelManager.java:113)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:243)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:222)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:174)
at org.apache.seata.core.rpc.netty.AbstractNettyRemotingClient.lambda$init$0(AbstractNettyRemotingClient.java:122)
at com.alibaba.ttl.TtlRunnable.run(TtlRunnable.java:60)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.seata.common.exception.FrameworkException: connect failed, can not connect to services-server.
at org.apache.seata.core.rpc.netty.NettyClientBootstrap.getNewChannel(NettyClientBootstrap.java:188)
... 17 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: Host is unreachable: /10.10.168.230:10222
Caused by: java.net.NoRouteToHostException: Host is unreachable
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-02-04 18:49:58,518 [timeoutChecker_1_1] ERROR [] [] [o.a.seata.core.rpc.netty.NettyClientChannelManager] - 0101 can not connect to [10.10.240.0:10222, 10.10.168.230:10222] cause:[can not register RM,err:can not connect to services-server.]
2026-02-04 18:49:58,518 [timeoutChecker_1_1] ERROR [] [] [o.a.seata.core.rpc.netty.NettyClientChannelManager] - connect server failed. can not connect to [10.10.240.0:10222, 10.10.168.230:10222]
org.apache.seata.common.exception.FrameworkException: can not connect to [10.10.240.0:10222, 10.10.168.230:10222]
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:271)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.doReconnect(NettyClientChannelManager.java:222)
at org.apache.seata.core.rpc.netty.NettyClientChannelManager.reconnect(NettyClientChannelManager.java:174)
at org.apache.seata.core.rpc.netty.AbstractNettyRemotingClient.lambda$init$0(AbstractNettyRemotingClient.java:122)
at com.alibaba.ttl.TtlRunnable.run(TtlRunnable.java:60)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)`
### Ⅲ. Describe what you expected to happen
_No response_
### Ⅳ. How to reproduce it (as minimally and precisely as possible)
_No response_
### Ⅴ. Anything else we need to know?
_No response_
### Ⅵ. Environment
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "biedongbin (via GitHub)" <gi...@apache.org>.
biedongbin commented on issue #7986:
URL: https://github.com/apache/incubator-seata/issues/7986#issuecomment-3850578087
> Is 10.10.168.230:10222 the new node after the restart, or an existing (old) node? Also, do you have a reproducible demo for this?
是之前的节点， seata-server 重启之后 k8s 分配了新的地址， 初步分析是 心跳检测或者客户端的 订阅有问题，导致没有及时刷新本地地址缓存所以一直在尝试旧的地址
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "biedongbin (via GitHub)" <gi...@apache.org>.
biedongbin commented on issue #7986:
URL: https://github.com/apache/incubator-seata/issues/7986#issuecomment-3850759129
我们在尝试测试环境重现问题， 你有什么其他排查思路或者建议吗？没有找到有任何相关的配置参数之类的内容，只有一个心跳是否为true的配置还是默认开启的
> > > Is 10.10.168.230:10222 the new node after the restart, or an existing (old) node? Also, do you have a reproducible demo for this?
> >
> >
> > 是之前的节点， seata-server 重启之后 k8s 分配了新的地址， 初步分析是 心跳检测或者客户端的 订阅有问题，导致没有及时刷新本地地址缓存所以一直在尝试旧的地址
>
> Can this be reproduced in an offline (non-production) environment? Could it be that the Nacos server is a newer version than the client, causing the client not to push to the new server address?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "biedongbin (via GitHub)" <gi...@apache.org>.
biedongbin commented on issue #7986:
URL: https://github.com/apache/incubator-seata/issues/7986#issuecomment-3870643882
问题找到了， 有点坑。。。。
nacos 2.1.1的bug, nacos 主动推送的rpc 消息内部 丢失了 clusters，正常的微服务那套集群模式 用不到 clusters 所以一切正常
seata 这边使用到了 clusters ，地址缓存目标key 为 group_name@@server_name@@cluster_name 然后就对不上了。。
查了内存里面发现类似如下的数据，所以就坑爹了， 但是客户端定时Job主动轮训去查的逻辑还没定位到，猜测是类似的情况
```
arthas@1]$ vmtool --action getInstances --className com.alibaba.nacos.client.naming.cache.ServiceInfoHolder --express 'instances[1].getServiceInfoMap' -x 3
@ConcurrentHashMap[
@String[DEFAULT_GROUP@@seata-server@@default-group]:@ServiceInfo[
jsonFromServer=@String[{"name":"seata-server","groupName":"DEFAULT_GROUP","clusters":"default-group","cacheMillis":10000,"hosts":[{"instanceId":"100.77.106.222#10222#default-group#DEFAULT_GROUP@@seata-server","ip":"100.77.106.222","port":10222,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"default-group","serviceName":"DEFAULT_GROUP@@seata-server","metadata":{},"ipDeleteTimeout":30000,"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"instanceIdGenerator":"simple"},{"instanceId":"100.111.206.134#10222#default-group#DEFAULT_GROUP@@seata-server","ip":"100.111.206.134","port":10222,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"default-group","serviceName":"DEFAULT_GROUP@@seata-server","metadata":{},"ipDeleteTimeout":30000,"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"instanceIdGenerator":"simple"}],"lastRefTime":1770621517835,"checksum":"","allIPs":false,"reachProtectionThreshold":fa
lse,"valid":true}],
EMPTY=@String[],
ALL_IPS=@String[000--00-ALL_IPS--00--000],
SPLITER=@String[@@],
DEFAULT_CHARSET=@String[UTF-8],
name=@String[seata-server],
groupName=@String[DEFAULT_GROUP],
clusters=@String[default-group],
cacheMillis=@Long[10000],
hosts=@ArrayList[
@Instance[Instance{instanceId='100.77.106.222#10222#default-group#DEFAULT_GROUP@@seata-server', ip='100.77.106.222', port=10222, weight=1.0, healthy=true, enabled=true, ephemeral=true, clusterName='default-group', serviceName='DEFAULT_GROUP@@seata-server', metadata={}}],
@Instance[Instance{instanceId='100.111.206.134#10222#default-group#DEFAULT_GROUP@@seata-server', ip='100.111.206.134', port=10222, weight=1.0, healthy=true, enabled=true, ephemeral=true, clusterName='default-group', serviceName='DEFAULT_GROUP@@seata-server', metadata={}}],
],
lastRefTime=@Long[1770621517835],
checksum=@String[],
allIPs=@Boolean[false],
reachProtectionThreshold=@Boolean[false],
],
@String[DEFAULT_GROUP@@seata-server]:@ServiceInfo[
jsonFromServer=@String[{"name":"seata-server","groupName":"DEFAULT_GROUP","clusters":"","cacheMillis":10000,"hosts":[{"instanceId":"100.111.206.134#10222#default-group#DEFAULT_GROUP@@seata-server","ip":"100.111.206.134","port":10222,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"default-group","serviceName":"DEFAULT_GROUP@@seata-server","metadata":{},"ipDeleteTimeout":30000,"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"instanceIdGenerator":"simple"}],"lastRefTime":1770621778959,"checksum":"","allIPs":false,"reachProtectionThreshold":false,"valid":true}],
EMPTY=@String[],
ALL_IPS=@String[000--00-ALL_IPS--00--000],
SPLITER=@String[@@],
DEFAULT_CHARSET=@String[UTF-8],
name=@String[seata-server],
groupName=@String[DEFAULT_GROUP],
clusters=@String[],
cacheMillis=@Long[10000],
hosts=@ArrayList[
@Instance[Instance{instanceId='100.111.206.134#10222#default-group#DEFAULT_GROUP@@seata-server', ip='100.111.206.134', port=10222, weight=1.0, healthy=true, enabled=true, ephemeral=true, clusterName='default-group', serviceName='DEFAULT_GROUP@@seata-server', metadata={}}],
],
lastRefTime=@Long[1770621778959],
checksum=@String[],
allIPs=@Boolean[false],
reachProtectionThreshold=@Boolean[false],
],
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on issue #7986:
URL: https://github.com/apache/incubator-seata/issues/7986#issuecomment-3850587705
> > Is 10.10.168.230:10222 the new node after the restart, or an existing (old) node? Also, do you have a reproducible demo for this?
>
> 是之前的节点， seata-server 重启之后 k8s 分配了新的地址， 初步分析是 心跳检测或者客户端的 订阅有问题，导致没有及时刷新本地地址缓存所以一直在尝试旧的地址
Can this be reproduced in an offline (non-production) environment? Could it be that the Nacos server is a newer version than the client, causing the client not to push to the new server address?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on issue #7986:
URL: https://github.com/apache/incubator-seata/issues/7986#issuecomment-3850567422
Is 10.10.168.230:10222 the new node after the restart, or an existing (old) node? Also, do you have a reproducible demo for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [I] 生产事故- seata-server 重启之后，客户端未能自动重连 [incubator-seata]
Posted by "biedongbin (via GitHub)" <gi...@apache.org>.
biedongbin closed issue #7986: 生产事故- seata-server 重启之后，客户端未能自动重连
URL: https://github.com/apache/incubator-seata/issues/7986
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org