You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ratis.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2026/10/02 16:01:19 UTC
[PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
jojochuang opened a new pull request, #1624:
URL: https://github.com/apache/ratis/pull/1624
## Jira
https://issues.apache.org/jira/browse/RATIS-2729
## Problem
Since RATIS-872, every leader-bound `RaftClient` request calls `RepliedCallIds.get(callId)`, which inserts into the `sent` map. On reply, `RepliedCallIds.add()` runs only for non-read-only requests, which removes the entry from `sent`.
Read-only RPC types (including `WATCH`) never call `add()`, so each completed watch leaves a permanent entry in `sent`. Long-lived `RaftClient` instances with high write + watch volume (for example Apache Ozone `XceiverClientRatis.watchForCommit()`) can accumulate millions of entries and OOM.
## Fix
For read-only replies, remove the outgoing callId from `sent` via `removeSent()` without touching the write `replied` queue used for RATIS-872 retry invalidation.
## Test plan
- [x] `mvn install -DskipTests` in ratis repo
- [x] Standalone repro (Ozone `dev-support/replied-call-ids-leak-repro/`): patched Ratis keeps `sent` size at 0 after 100 write+watch pairs; unpatched 3.3.1 grows to 101 with `-DexpectLeak=true -Dratis.version=3.3.1`
Generated-by: Cursor (Auto)
Made with [Cursor](https://cursor.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #1624:
URL: https://github.com/apache/ratis/pull/1624#discussion_r4167818225
##########
ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java:
##########
@@ -362,6 +367,8 @@ RaftClientReply handleReply(RaftClientRequest request, RaftClientReply reply) {
if (request.isToLeader() && reply != null) {
if (!request.getType().isReadOnly()) {
repliedCallIds.add(reply.getCallId());
+ } else {
+ repliedCallIds.removeSent(request.getCallId());
Review Comment:
If we removeSent, the corresponding reply id set will be ignored. Let's don't put it to `sent` at the first place.
(below is revised)
```diff
+++ b/ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java
@@ -155,13 +155,15 @@ public final class RaftClientImpl implements RaftClient {
/** @return the replied callIds for the given callId. */
Iterable<Long> get(long callId) {
- final Supplier<Set<Long>> supplier = MemoizedSupplier.valueOf(this::getAndReset);
- final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> supplier.get()));
+ final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> getAndReset()));
LOG.debug("{}: get {} returns {}", name, callId, set);
return set;
}
- private synchronized Set<Long> getAndReset() {
+ synchronized Set<Long> getAndReset() {
+ if (replied.isEmpty()) {
+ return Collections.emptySet();
+ }
final Set<Long> previous = replied;
replied = new TreeSet<>();
return previous;
@@ -290,7 +292,7 @@ public final class RaftClientImpl implements RaftClient {
b.setServerId(server);
} else {
b.setLeaderId(getLeaderId())
- .setRepliedCallIds(repliedCallIds.get(callId));
+ .setRepliedCallIds(type.isReadOnly() ? repliedCallIds.getAndReset() : repliedCallIds.get(callId));
}
if (TraceUtils.isEnabled()) {
b.setSpanContext(TraceUtils.injectContextToProto());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #1624:
URL: https://github.com/apache/ratis/pull/1624#discussion_r4167818225
##########
ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java:
##########
@@ -362,6 +367,8 @@ RaftClientReply handleReply(RaftClientRequest request, RaftClientReply reply) {
if (request.isToLeader() && reply != null) {
if (!request.getType().isReadOnly()) {
repliedCallIds.add(reply.getCallId());
+ } else {
+ repliedCallIds.removeSent(request.getCallId());
Review Comment:
If we removeSent, the corresponding reply id set will be ignored. Let's don't put it to `sent` at the first place.
(below is revised)
```diff
@@ -290,7 +292,7 @@ public final class RaftClientImpl implements RaftClient {
b.setServerId(server);
} else {
b.setLeaderId(getLeaderId())
- .setRepliedCallIds(repliedCallIds.get(callId));
+ .setRepliedCallIds(type.isReadOnly() ? repliedCallIds.getAndReset() : repliedCallIds.get(callId));
}
if (TraceUtils.isEnabled()) {
b.setSpanContext(TraceUtils.injectContextToProto());
```
BTW, let's also improve the code a little bit: Not to create a supplier and not to create a new set for empty set.
```diff
+++ b/ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java
@@ -155,13 +155,15 @@ public final class RaftClientImpl implements RaftClient {
/** @return the replied callIds for the given callId. */
Iterable<Long> get(long callId) {
- final Supplier<Set<Long>> supplier = MemoizedSupplier.valueOf(this::getAndReset);
- final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> supplier.get()));
+ final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> getAndReset()));
LOG.debug("{}: get {} returns {}", name, callId, set);
return set;
}
- private synchronized Set<Long> getAndReset() {
+ synchronized Set<Long> getAndReset() {
+ if (replied.isEmpty()) {
+ return Collections.emptySet();
+ }
final Set<Long> previous = replied;
replied = new TreeSet<>();
return previous;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #1624:
URL: https://github.com/apache/ratis/pull/1624#discussion_r4169067703
##########
ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java:
##########
@@ -362,6 +367,8 @@ RaftClientReply handleReply(RaftClientRequest request, RaftClientReply reply) {
if (request.isToLeader() && reply != null) {
if (!request.getType().isReadOnly()) {
repliedCallIds.add(reply.getCallId());
+ } else {
+ repliedCallIds.removeSent(request.getCallId());
Review Comment:
Appreciate. Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #1624:
URL: https://github.com/apache/ratis/pull/1624#discussion_r4167818225
##########
ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java:
##########
@@ -362,6 +367,8 @@ RaftClientReply handleReply(RaftClientRequest request, RaftClientReply reply) {
if (request.isToLeader() && reply != null) {
if (!request.getType().isReadOnly()) {
repliedCallIds.add(reply.getCallId());
+ } else {
+ repliedCallIds.removeSent(request.getCallId());
Review Comment:
If we removeSent, the corresponding reply id set will be ignored. Let's don't add it at the first place.
```diff
+++ b/ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java
@@ -289,8 +289,10 @@ public final class RaftClientImpl implements RaftClient {
if (server != null) {
b.setServerId(server);
} else {
- b.setLeaderId(getLeaderId())
- .setRepliedCallIds(repliedCallIds.get(callId));
+ b.setLeaderId(getLeaderId());
+ if (!type.isReadOnly()) {
+ b.setRepliedCallIds(repliedCallIds.get(callId));
+ }
}
if (TraceUtils.isEnabled()) {
b.setSpanContext(TraceUtils.injectContextToProto());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #1624:
URL: https://github.com/apache/ratis/pull/1624
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RATIS-2729. Remove RepliedCallIds.sent entries for read-only replies [ratis]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #1624:
URL: https://github.com/apache/ratis/pull/1624#discussion_r4167818225
##########
ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java:
##########
@@ -362,6 +367,8 @@ RaftClientReply handleReply(RaftClientRequest request, RaftClientReply reply) {
if (request.isToLeader() && reply != null) {
if (!request.getType().isReadOnly()) {
repliedCallIds.add(reply.getCallId());
+ } else {
+ repliedCallIds.removeSent(request.getCallId());
Review Comment:
If we removeSent, the corresponding reply id set will be ignored. Let's don't put it to `sent` at the first place.
(below is revised)
```diff
@@ -290,7 +292,7 @@ public final class RaftClientImpl implements RaftClient {
b.setServerId(server);
} else {
b.setLeaderId(getLeaderId())
- .setRepliedCallIds(repliedCallIds.get(callId));
+ .setRepliedCallIds(type.isReadOnly() ? repliedCallIds.getAndReset() : repliedCallIds.get(callId));
}
if (TraceUtils.isEnabled()) {
b.setSpanContext(TraceUtils.injectContextToProto());
```
BTW, let's also improve the code a little bit:
```diff
+++ b/ratis-client/src/main/java/org/apache/ratis/client/impl/RaftClientImpl.java
@@ -155,13 +155,15 @@ public final class RaftClientImpl implements RaftClient {
/** @return the replied callIds for the given callId. */
Iterable<Long> get(long callId) {
- final Supplier<Set<Long>> supplier = MemoizedSupplier.valueOf(this::getAndReset);
- final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> supplier.get()));
+ final Set<Long> set = Collections.unmodifiableSet(sent.computeIfAbsent(callId, cid -> getAndReset()));
LOG.debug("{}: get {} returns {}", name, callId, set);
return set;
}
- private synchronized Set<Long> getAndReset() {
+ synchronized Set<Long> getAndReset() {
+ if (replied.isEmpty()) {
+ return Collections.emptySet();
+ }
final Set<Long> previous = replied;
replied = new TreeSet<>();
return previous;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ratis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org