You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "zhengchenyu (via GitHub)" <gi...@apache.org> on 2026/06/12 03:49:34 UTC
[PR] YARN-11965. Support partition-aware resource metrics in RM cluster me… [hadoop]
zhengchenyu opened a new pull request, #8547:
URL: https://github.com/apache/hadoop/pull/8547
### Description of PR
When node labels are enabled, different labels represent separate resource pools. However, the RM REST API `/ws/v1/cluster/metrics` currently exposes fields such as totalMB and totalVirtualCores based on the default partition only. As a result, resources from non-default partitions are not visible to external resource management systems, which may incorrectly determine that the cluster has no available capacity after new labels are added.
The API should expose cluster resource metrics across all partitions and provide partition-level metrics so clients can distinguish capacity and usage by node label.
### How was this patch tested?
unit test and test in cluster based on internal hadoop version.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4705229462
After this PR, curl `http://{rm-host}:{rm-port/ws/v1/cluster/metrics` will fix the issue where parameters such as `totalMB` and `totalVirtualCores` are cluster-specific, instead of the default label parameters. Additionally, `partitionClusterMetrics` has been added to represent metrics for different partitions. The updated results are shown below:
```
{
"clusterMetrics": {
"appsSubmitted": 15,
"appsCompleted": 13,
"appsPending": 1,
"appsRunning": 1,
"appsFailed": 0,
"appsKilled": 0,
"reservedMB": 0,
"availableMB": 840704,
"allocatedMB": 183296,
"pendingMB": 5983232,
"reservedVirtualCores": 0,
"availableVirtualCores": 188,
"allocatedVirtualCores": 60,
"pendingVirtualCores": 1948,
"aggregateContainersAllocated": 214,
"aggregateContainersReleased": 199,
"containersAllocated": 15,
"containersReserved": 0,
"containersPending": 487,
"totalMB": 1024000,
"totalVirtualCores": 248,
"totalNodes": 4,
"lostNodes": 0,
"unhealthyNodes": 0,
"unscheduledNodes": 0,
"decommissioningNodes": 0,
"decommissionedNodes": 0,
"rebootedNodes": 0,
"activeNodes": 4,
"shutdownNodes": 0,
"totalUsedResourcesAcrossPartition": {
"memory": 183296,
"vCores": 60,
"resourceInformations": {
"resourceInformation": [
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "memory-mb",
"resourceType": "COUNTABLE",
"units": "Mi",
"value": 183296
},
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "vcores",
"resourceType": "COUNTABLE",
"units": "",
"value": 60
}
]
}
},
"totalClusterResourcesAcrossPartition": {
"memory": 1024000,
"vCores": 248,
"resourceInformations": {
"resourceInformation": [
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "memory-mb",
"resourceType": "COUNTABLE",
"units": "Mi",
"value": 1024000
},
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "vcores",
"resourceType": "COUNTABLE",
"units": "",
"value": 248
}
]
}
},
"totalReservedResourcesAcrossPartition": {
"memory": 0,
"vCores": 0,
"resourceInformations": {
"resourceInformation": [
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "memory-mb",
"resourceType": "COUNTABLE",
"units": "Mi",
"value": 0
},
{
"maximumAllocation": 9223372036854775807,
"minimumAllocation": 0,
"name": "vcores",
"resourceType": "COUNTABLE",
"units": "",
"value": 0
}
]
}
},
"totalAllocatedContainersAcrossPartition": 15,
"partitionClusterMetrics": [
{
"partitionName": "",
"totalMB": 256000,
"totalVirtualCores": 62,
"availableMB": 72704,
"availableVirtualCores": 2,
"allocatedMB": 183296,
"allocatedVirtualCores": 60,
"reservedMB": 0,
"reservedVirtualCores": 0,
"pendingMB": 5983232,
"pendingVirtualCores": 1948,
"containersAllocated": 15,
"containersReserved": 0,
"containersPending": 487
},
{
"partitionName": "part-a",
"totalMB": 256000,
"totalVirtualCores": 62,
"availableMB": 256000,
"availableVirtualCores": 62,
"allocatedMB": 0,
"allocatedVirtualCores": 0,
"reservedMB": 0,
"reservedVirtualCores": 0,
"pendingMB": 0,
"pendingVirtualCores": 0,
"containersAllocated": 0,
"containersReserved": 0,
"containersPending": 0
},
{
"partitionName": "part-b",
"totalMB": 256000,
"totalVirtualCores": 62,
"availableMB": 256000,
"availableVirtualCores": 62,
"allocatedMB": 0,
"allocatedVirtualCores": 0,
"reservedMB": 0,
"reservedVirtualCores": 0,
"pendingMB": 0,
"pendingVirtualCores": 0,
"containersAllocated": 0,
"containersReserved": 0,
"containersPending": 0
},
{
"partitionName": "part-c",
"totalMB": 256000,
"totalVirtualCores": 62,
"availableMB": 256000,
"availableVirtualCores": 62,
"allocatedMB": 0,
"allocatedVirtualCores": 0,
"reservedMB": 0,
"reservedVirtualCores": 0,
"pendingMB": 0,
"pendingVirtualCores": 0,
"containersAllocated": 0,
"containersReserved": 0,
"containersPending": 0
}
],
"crossPartitionMetricsAvailable": true
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "K0K0V0K (via GitHub)" <gi...@apache.org>.
K0K0V0K commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4709807027
Thanks @zhengchenyu for the change!
Overall, this looks good to me.
One thing that came to mind: if I understand correctly, the `/ws/v1/cluster/metrics` endpoint currently has a fairly predictable response size and response time. With this change, both could become dependent on the number of partitions in the cluster.
As far as I know, a single ResourceManager can handle around 10,000 NodeManagers. In a hypothetical worst-case scenario where each NodeManager belongs to a different partition, could you measure how the response time is affected?
I’m not familiar with your use case, but have you considered exposing this information through a separate endpoint instead? For example, something like `/ws/v1/cluster/partitions/metrics` or` /ws/v1/cluster/partition-metrics` might avoid introducing any regression in the response size or latency of the existing /ws/v1/cluster/metrics endpoint while still making the additional data available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#discussion_r3414873799
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java:
##########
@@ -1275,6 +1275,18 @@ public int getReservedVirtualCores() {
public int getReservedContainers() {
return reservedContainers.value();
}
+
+ public int getAllocatedContainers(String partition) {
+ return getPartitionMetrics(partition).getAllocatedContainers();
+ }
+
+ public int getPendingContainers(String partition) {
+ return getPartitionMetrics(partition).getPendingContainers();
+ }
+
+ public int getReservedContainers(String partition) {
+ return getPartitionMetrics(partition).getReservedContainers();
+ }
Review Comment:
The new partition-specific container getters call `getPartitionMetrics(partition)`, which lazily creates and registers a `PartitionQueueMetrics` instance when it doesn't already exist. Since `/ws/v1/cluster/metrics` calls these methods for every known partition, a read-only REST request can mutate the metrics registry and potentially grow the number of registered metric sources over time. Prefer a non-creating lookup (return 0 when missing) to keep the REST path side-effect free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4842708064
@K0K0V0K @slfan1989 How about merging this PR or further review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster me… [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4688119693
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 27m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 55s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 30s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 58 unchanged - 0 fixed = 59 total (was 58) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 22s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 91m 40s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not generate ASF License warnings. |
| | | 165m 1s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8547 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d66ab5fae701 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 99f2c5225f0d55c9e50c412345a4a162beb4429d |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/1/testReport/ |
| Max. process+thread count | 977 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/1/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4714141971
@K0K0V0K
On a cluster of 10,000+ nodes, having one partition per node is not a practical use case, as it would significantly increase the operational burden. Generally, there are only a few NodeLabels, and each partition only has 14 fields, so I don't think it will create a significant burden.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4770432778
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 28m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 32s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 58 unchanged - 0 fixed = 61 total (was 58) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 91m 40s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 172m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.55 ServerAPI=1.55 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8547 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 92d0650421af 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a088aab15d5c0ab4692c998a387d06d97b0306ac |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/3/testReport/ |
| Max. process+thread count | 980 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/3/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#discussion_r3418678094
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java:
##########
@@ -338,6 +338,47 @@ public void testSchedulerPartitionsXML(boolean pLegacyQueueMode)
verifySchedulerInfoXML(dom);
}
+ @MethodSource("getParameters")
+ @ParameterizedTest(name = "{index}: legacy-queue-mode={0}")
+ public void testClusterMetricsWithPartitions(boolean pLegacyQueueMode)
+ throws Exception {
+ initTestRMWebServicesForCSWithPartitions(pLegacyQueueMode);
+ WebTarget r = targetWithJsonObject();
+ Response response = r.path("ws").path("v1").path("cluster")
+ .path("metrics").request(MediaType.APPLICATION_JSON).get(Response.class);
Review Comment:
Added the unit test for xml response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4705235749
@slfan1989 @K0K0V0K Can you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#issuecomment-4717115802
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 43s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 26m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | compile | 0m 56s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 12s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 29s | [/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 58 unchanged - 0 fixed = 59 total (was 58) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK Ubuntu-21.0.11+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 91m 42s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not generate ASF License warnings. |
| | | 171m 5s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8547 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b5511995a369 5.15.0-179-generic #189-Ubuntu SMP Tue May 5 18:20:56 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f52dda14b101cd29211b8849458ff8e738620f15 |
| Default Java | Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.11+10-1-24.04.2-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.19+10-1-24.04.2-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/2/testReport/ |
| Max. process+thread count | 935 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8547/2/console |
| versions | git=2.43.0 maven=3.9.15 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#discussion_r3414873841
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesForCSWithPartitions.java:
##########
@@ -338,6 +338,47 @@ public void testSchedulerPartitionsXML(boolean pLegacyQueueMode)
verifySchedulerInfoXML(dom);
}
+ @MethodSource("getParameters")
+ @ParameterizedTest(name = "{index}: legacy-queue-mode={0}")
+ public void testClusterMetricsWithPartitions(boolean pLegacyQueueMode)
+ throws Exception {
+ initTestRMWebServicesForCSWithPartitions(pLegacyQueueMode);
+ WebTarget r = targetWithJsonObject();
+ Response response = r.path("ws").path("v1").path("cluster")
+ .path("metrics").request(MediaType.APPLICATION_JSON).get(Response.class);
Review Comment:
`/ws/v1/cluster/metrics` supports both JSON and XML responses, but the new partition-aware fields are only covered by a JSON test here. Adding an XML variant would help ensure JAXB/MOXy marshalling of `partitionClusterMetrics` and the cross-partition resource fields remains compatible for XML clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-11965. Support partition-aware resource metrics in RM cluster metrics REST API. [hadoop]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #8547:
URL: https://github.com/apache/hadoop/pull/8547#discussion_r3418533730
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java:
##########
@@ -1275,6 +1275,18 @@ public int getReservedVirtualCores() {
public int getReservedContainers() {
return reservedContainers.value();
}
+
+ public int getAllocatedContainers(String partition) {
+ return getPartitionMetrics(partition).getAllocatedContainers();
+ }
+
+ public int getPendingContainers(String partition) {
+ return getPartitionMetrics(partition).getPendingContainers();
+ }
+
+ public int getReservedContainers(String partition) {
+ return getPartitionMetrics(partition).getReservedContainers();
+ }
Review Comment:
This is a good suggestion to avoid creating partition metrics unnecessarily. I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org