You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@seata.apache.org by "saul-c (via GitHub)" <gi...@apache.org> on 2026/09/12 03:11:06 UTC
[PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
saul-c opened a new pull request, #8224:
URL: https://github.com/apache/incubator-seata/pull/8224
When TM/RM clients disconnect (e.g. K8s pod rolling update, HPA scaling), RpcContext.release() only nulled the clientIDHolderMap field instead of removing the dead channel from ChannelManager.IDENTIFIED_CHANNELS. Dead Netty channels (with RpcContext/pipeline/ByteBuf) accumulated forever until the TC ran into continuous FullGC and OOM.
- RpcContext.release(): remove the channel from IDENTIFIED_CHANNELS
- ChannelManager.releaseRpcContext(): remove the channel defensively
- AbstractNettyRemotingServer.handleDisconnect: route through the unified releaseRpcContext entry (used by both READER_IDLE and channelInactive)
- NettyClientChannelManager.releaseChannel(): release rpc context too
- add ChannelManagerOOMTest reproducing repeated reconnect/disconnect
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Please make sure you have read and understood the contributing guidelines -->
- [ ] I have read the [CONTRIBUTING.md](https://github.com/apache/incubator-seata/blob/2.x/CONTRIBUTING.md) guidelines.
- [ ] I have registered the PR [changes](https://github.com/apache/incubator-seata/tree/2.x/changes).
### Ⅰ. Describe what this PR did
### Ⅱ. Does this pull request fix one issue?
<!-- If that, add "fixes #xxx" below in the next line, for example, fixes #97. -->
### Ⅲ. Why don't you add test cases (unit test/integration test)?
### Ⅳ. Describe how to verify it
### Ⅴ. Special notes for reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
Posted by "WangzJi (via GitHub)" <gi...@apache.org>.
WangzJi merged PR #8224:
URL: https://github.com/apache/incubator-seata/pull/8224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
Posted by "YongGoose (via GitHub)" <gi...@apache.org>.
YongGoose commented on PR #8224:
URL: https://github.com/apache/incubator-seata/pull/8224#issuecomment-6060818617
@saul-c This is a good PR. I think it could be ready to merge with just a few more tweaks. Would you have some time to take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
Posted by "YongGoose (via GitHub)" <gi...@apache.org>.
YongGoose commented on code in PR #8224:
URL: https://github.com/apache/incubator-seata/pull/8224#discussion_r4167214836
##########
core/src/test/java/org/apache/seata/core/rpc/ChannelManagerOOMTest.java:
##########
@@ -0,0 +1,210 @@
+/*
+ * Copyright 1999-2019 Seata.io Group.
Review Comment:
Please use the ASF license header and drop the `@author` tag like the rest of the project; otherwise the license check will fail.
##########
core/src/main/java/org/apache/seata/core/rpc/RpcContext.java:
##########
@@ -74,6 +74,7 @@ public class RpcContext {
public void release() {
Integer clientPort = ChannelUtil.getClientPortFromChannel(channel);
if (clientIDHolderMap != null) {
+ clientIDHolderMap.remove(channel);
Review Comment:
Removing the context from `IDENTIFIED_CHANNELS` here breaks the TM fallback in `ChannelManager.getSameClientChannel()`: for an inactive channel it now returns null, so `sendAsyncResponse` throws `channel is error.` instead of replying on the TM's other live channel.
##########
core/src/main/java/org/apache/seata/core/rpc/netty/ChannelManager.java:
##########
@@ -287,6 +287,7 @@ public static void releaseRpcContext(Channel channel) {
RpcContext rpcContext = getContextFromIdentified(channel);
if (rpcContext != null) {
rpcContext.release();
+ IDENTIFIED_CHANNELS.remove(channel);
Review Comment:
`exceptionCaught` calls `releaseRpcContext` without closing the channel, so a still-open channel loses its context and the next `MergedWarpMessage` throws an NPE at `rpcContext.getVersion()` in `processMessage`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #8224:
URL: https://github.com/apache/incubator-seata/pull/8224#issuecomment-5740685686
## [Codecov](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 73.35%. Comparing base ([`41694e1`](https://app.codecov.io/gh/apache/incubator-seata/commit/41694e140b40894aa3aae4cfe6ecffed9b321a36?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`bb43d4a`](https://app.codecov.io/gh/apache/incubator-seata/commit/bb43d4a119072caa804ebd22f719642e5ee286a1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## 2.x #8224 +/- ##
============================================
+ Coverage 73.22% 73.35% +0.12%
Complexity 1146 1146
============================================
Files 1153 1153
Lines 42348 42352 +4
Branches 5061 5061
============================================
+ Hits 31011 31068 +57
+ Misses 8852 8807 -45
+ Partials 2485 2477 -8
```
| [Files with missing lines](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ain/java/org/apache/seata/core/rpc/RpcContext.java](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?src=pr&el=tree&filepath=core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseata%2Fcore%2Frpc%2FRpcContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2VhdGEvY29yZS9ycGMvUnBjQ29udGV4dC5qYXZh) | `93.82% <100.00%> (+0.07%)` | :arrow_up: |
| [...ta/core/rpc/netty/AbstractNettyRemotingServer.java](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?src=pr&el=tree&filepath=core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseata%2Fcore%2Frpc%2Fnetty%2FAbstractNettyRemotingServer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2VhdGEvY29yZS9ycGMvbmV0dHkvQWJzdHJhY3ROZXR0eVJlbW90aW5nU2VydmVyLmphdmE=) | `88.88% <100.00%> (+0.09%)` | :arrow_up: |
| [...rg/apache/seata/core/rpc/netty/ChannelManager.java](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?src=pr&el=tree&filepath=core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseata%2Fcore%2Frpc%2Fnetty%2FChannelManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2VhdGEvY29yZS9ycGMvbmV0dHkvQ2hhbm5lbE1hbmFnZXIuamF2YQ==) | `70.14% <100.00%> (+0.48%)` | :arrow_up: |
| [...eata/core/rpc/netty/NettyClientChannelManager.java](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?src=pr&el=tree&filepath=core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fseata%2Fcore%2Frpc%2Fnetty%2FNettyClientChannelManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2VhdGEvY29yZS9ycGMvbmV0dHkvTmV0dHlDbGllbnRDaGFubmVsTWFuYWdlci5qYXZh) | `84.53% <100.00%> (+0.08%)` | :arrow_up: |
... and [18 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-seata/pull/8224/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
[](https://app.codecov.io/gh/apache/incubator-seata/pull/8224?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org
Re: [PR] bugfix: fix IDENTIFIED_CHANNELS leak causing TC FullGC and OOM [incubator-seata]
Posted by "funky-eyes (via GitHub)" <gi...@apache.org>.
funky-eyes commented on code in PR #8224:
URL: https://github.com/apache/incubator-seata/pull/8224#discussion_r4078732172
##########
core/src/main/java/org/apache/seata/core/rpc/netty/AbstractNettyRemotingServer.java:
##########
@@ -215,6 +215,7 @@ private void handleDisconnect(ChannelHandlerContext ctx) {
}
if (rpcContext != null && rpcContext.getClientRole() != null) {
rpcContext.release();
+ ChannelManager.releaseRpcContext(ctx.channel());
Review Comment:
If the server happens to need this channel to send a response at that moment, releasing it prematurely could result in an NPE. Therefore, I believe there should be a delayed cleanup mechanism to ensure that the channel is only released once the server can no longer reference or use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@seata.apache.org
For additional commands, e-mail: notifications-help@seata.apache.org