You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "duongkame (via GitHub)" <gi...@apache.org> on 2023/10/11 00:43:10 UTC
[PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
duongkame opened a new pull request, #5420:
URL: https://github.com/apache/ozone/pull/5420
This metric shows how many threads are really working in an RPC server.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9424
## How was this patch tested?
Tested locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1763144557
Filed HDDS-9454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #5420:
URL: https://github.com/apache/ozone/pull/5420#discussion_r1355728528
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java:
##########
@@ -83,11 +83,15 @@ public RESPONSE processRequest(
}
long startTime = System.currentTimeMillis();
-
- RESPONSE response = methodCall.apply(request);
-
- protocolMessageMetrics.increment(type,
- System.currentTimeMillis() - startTime);
+ protocolMessageMetrics.increaseConcurrency();
+ RESPONSE response;
+ try {
+ response = methodCall.apply(request);
+ } finally {
+ protocolMessageMetrics.increment(type,
+ System.currentTimeMillis() - startTime);
+ protocolMessageMetrics.decreaseConcurrency();
+ }
Review Comment:
Let's use a `org.apache.ratis.util.UncheckedAutoCloseable`, i.e.
```java
//processRequest
final RESPONSE response;
try (UncheckedAutoCloseable ignored = protocolMessageMetrics.measure(type)) {
response = methodCall.apply(request);
}
```
```java
//ProtocolMessageMetrics
public UncheckedAutoCloseable measure(KEY key) {
final long startTime = System.currentTimeMillis();
concurrency.incrementAndGet();
return () -> {
concurrency.decrementAndGet();
counters.get(key).incrementAndGet();
elapsedTimes.get(key).addAndGet(System.currentTimeMillis() - startTime);
};
}
```
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/ProtocolMessageMetrics.java:
##########
@@ -44,6 +45,8 @@ public class ProtocolMessageMetrics<KEY> implements MetricsSource {
private Map<KEY, AtomicLong> elapsedTimes =
new ConcurrentHashMap<>();
+ private AtomicInteger concurrency = new AtomicInteger(0);
Review Comment:
Let's make it `final`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on code in PR #5420:
URL: https://github.com/apache/ozone/pull/5420#discussion_r1358662517
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/OzoneProtocolMessageDispatcher.java:
##########
@@ -83,11 +83,15 @@ public RESPONSE processRequest(
}
long startTime = System.currentTimeMillis();
-
- RESPONSE response = methodCall.apply(request);
-
- protocolMessageMetrics.increment(type,
- System.currentTimeMillis() - startTime);
+ protocolMessageMetrics.increaseConcurrency();
+ RESPONSE response;
+ try {
+ response = methodCall.apply(request);
+ } finally {
+ protocolMessageMetrics.increment(type,
+ System.currentTimeMillis() - startTime);
+ protocolMessageMetrics.decreaseConcurrency();
+ }
Review Comment:
Good idea, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1763143227
> https://github.com/apache/ozone/actions/runs/6511750255/job/17688866092?pr=5420#step:5:3787
@adoroszlai , is there a way to download the test output log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1763293749
> is there a way to download the test output log?
Yes, `it-filesystem` from https://github.com/apache/ozone/actions/runs/6511750255?pr=5420#artifacts has the logs. (Attached it to HDDS-9454.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #5420:
URL: https://github.com/apache/ozone/pull/5420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1756533513
cc. @tanvipenumudy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1762884190
Leak reported from `TestRootedOzoneFileSystem`:
```
2023-10-13 19:29:21,163 [Finalizer] WARN db.CodecBuffer (CodecBuffer.java:detectLeaks(278)) - LEAK 1: [0<=32<=96: 2F 76 6F 6C 75 6D 65 38 33 39 ...], refCnt=1, capacity=96 allocation:
org.apache.hadoop.hdds.utils.db.CodecBuffer.<init>(CodecBuffer.java:53)
org.apache.hadoop.hdds.utils.db.CodecBuffer$LeakDetector$1.<init>(CodecBuffer.java:77)
org.apache.hadoop.hdds.utils.db.CodecBuffer$LeakDetector.newCodecBuffer(CodecBuffer.java:77)
org.apache.hadoop.hdds.utils.db.CodecBuffer$Factory.newCodecBuffer(CodecBuffer.java:70)
org.apache.hadoop.hdds.utils.db.CodecBuffer$Factory.newCodecBuffer(CodecBuffer.java:66)
org.apache.hadoop.hdds.utils.db.CodecBuffer.allocate(CodecBuffer.java:195)
org.apache.hadoop.hdds.utils.db.CodecBuffer$Allocator$1.apply(CodecBuffer.java:147)
org.apache.hadoop.hdds.utils.db.CodecBuffer$Allocator$1.apply(CodecBuffer.java:144)
org.apache.hadoop.hdds.utils.db.StringCodecBase.toCodecBuffer(StringCodecBase.java:174)
org.apache.hadoop.hdds.utils.db.StringCodec.toCodecBuffer(StringCodec.java:28)
org.apache.hadoop.hdds.utils.db.StringCodecBase.toCodecBuffer(StringCodecBase.java:42)
org.apache.hadoop.hdds.utils.db.Codec.toDirectCodecBuffer(Codec.java:73)
org.apache.hadoop.hdds.utils.db.TypedTable.encodeKeyCodecBuffer(TypedTable.java:132)
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:416)
org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:55)
org.apache.hadoop.ozone.om.KeyManagerImpl.createFakeDirIfShould(KeyManagerImpl.java:1273)
org.apache.hadoop.ozone.om.KeyManagerImpl.getOzoneFileStatus(KeyManagerImpl.java:1183)
org.apache.hadoop.ozone.om.KeyManagerImpl.getFileStatus(KeyManagerImpl.java:1145)
org.apache.hadoop.ozone.om.KeyManagerImpl.getFileStatus(KeyManagerImpl.java:1120)
org.apache.hadoop.ozone.om.TrashOzoneFileSystem.getFileStatus(TrashOzoneFileSystem.java:280)
org.apache.hadoop.ozone.om.TrashOzoneFileSystem.exists(TrashOzoneFileSystem.java:336)
org.apache.hadoop.ozone.om.TrashOzoneFileSystem.getTrashRoots(TrashOzoneFileSystem.java:316)
org.apache.hadoop.ozone.om.TrashPolicyOzone$Emptier.run(TrashPolicyOzone.java:317)
```
https://github.com/apache/ozone/actions/runs/6511750255/job/17688866092?pr=5420#step:5:3787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1763124046
@adoroszlai , thanks for letting me know the leakage. It looks like that the db is closed before calling `iterate(prefix)`. The prefix buffer is not released in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9424. Add a metric to indicate how many concurrent requests an RPC Server (OM, SCM) is handling [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5420:
URL: https://github.com/apache/ozone/pull/5420#issuecomment-1763357988
Thanks @duongkame for the patch, @szetszwo for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org