You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ss77892 (via GitHub)" <gi...@apache.org> on 2026/09/12 14:47:37 UTC
[PR] HDDS-16407. In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree [ozone]
ss77892 opened a new pull request, #11234:
URL: https://github.com/apache/ozone/pull/11234
## What changes were proposed in this pull request?
In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree
OMRatisHelper.convertResponseToMessage returned a lambda Message that captured the parsed OMResponse and serialized it on every getContent() call. Ratis keeps the reply Message of every applied write in its retry cache, on the leader and on every follower, for the retry cache timeout, and the cache has no size bound. So under a write burst each OM retained the whole parsed response tree (KeyInfo, locations, ACLs) per request for that long, and re-serialized it each time the content was read.
Serialize once, up front, and hand Ratis a Message that holds only the bytes. Same bytes on the wire and in the cache, so retried requests are still answered from the retry cache exactly as before.
Comparison to master on freon omkg workload with 1m keys. OM instances have 3gb heap size:
||instance||metric||09\-11 16:22 · HDDS\-16407@c750bd76e8||09\-07 22:11 · master@54da65a9d4||
|node1|gc\_count|645.0 (\-33.2%)|965.0|
|node2|gc\_count|1,329 (\+39.2%)|955.0|
|node3|gc\_count|648.0 (\-75.0%)|2,589|
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16407
## How was this patch tested?
UT
Benchmark on 3 node cluster using freon omkg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16407. In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree [ozone]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #11234:
URL: https://github.com/apache/ozone/pull/11234#discussion_r4000304403
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OMRatisHelper.java:
##########
@@ -53,9 +53,16 @@ public static OMRequest convertByteStringToOMRequest(ByteString bytes) throws IO
return OMRequest.parseFrom(new ByteBufferInputStream(buffer));
}
- /** Convert the given proto 2 response to a proto 3 {@link ByteString}. */
+ /**
+ * Convert the given proto 2 response to a proto 3 {@link ByteString}.
Review Comment:
nit: please change “proto 3 `ByteString`” to “Ratis `Message`”, since this method returns `Message`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16407. In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel merged PR #11234:
URL: https://github.com/apache/ozone/pull/11234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16407. In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #11234:
URL: https://github.com/apache/ozone/pull/11234#issuecomment-5734276418
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16407. In OM serialize the Ratis reply eagerly so the retry cache holds bytes, not the OMResponse tree [ozone]
Posted by "ss77892 (via GitHub)" <gi...@apache.org>.
ss77892 commented on code in PR #11234:
URL: https://github.com/apache/ozone/pull/11234#discussion_r4006040571
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OMRatisHelper.java:
##########
@@ -53,9 +53,16 @@ public static OMRequest convertByteStringToOMRequest(ByteString bytes) throws IO
return OMRequest.parseFrom(new ByteBufferInputStream(buffer));
}
- /** Convert the given proto 2 response to a proto 3 {@link ByteString}. */
+ /**
+ * Convert the given proto 2 response to a proto 3 {@link ByteString}.
Review Comment:
Done. Thank you for the review, @rich7420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org