You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hean-Chhinling (via GitHub)" <gi...@apache.org> on 2026/09/29 07:11:50 UTC

[PR] YARN-11996: Add RM JStack and Scheduler bulk-activities for diagnostic in Capacity Scheduler UI [hadoop]

Hean-Chhinling opened a new pull request, #8765:
URL: https://github.com/apache/hadoop/pull/8765

   Contains content generated by Cursor.
   
   ### Description of PR
   Extend the Diagnostic Feature in Capacity Scheduler UI with the followings 2 endpoints:
   
   1. JStacks for ResourceManager - `{rm_address}/ws/v1/cluster/jstack/{numberOfJStack}`
   2. Bulk activities - `{rm_address}/ws/v1/cluster/scheduler/bulk-activities`
   
   The two new endpoints could be seen at the last option here:
   <img width="507" height="720" alt="Screenshot 2026-09-29 at 13 50 08" src="https://github.com/user-attachments/assets/c1111a3b-0a18-4bbe-b6d2-05b17d737be5" />
   
   This PR depends on the following PR for getting the RM JStacks(by default, it is false and has to be turned on):
   https://github.com/apache/hadoop/pull/8123
   
   ### How was this patch tested?
   Tested locally and write unit-tests for it.
   
   ### For code changes:
   
   - [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint
         declared according to the connector-specific documentation? *Note: Automated CI
         testing doesn't cover all cases so manual testing with cloud storage is still
         required.*
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [ ] The PR includes the phrase "Contains content generated by <tool>"
         where <tool> is the name of the AI tool used.
   - [ ] My use of AI contributions follows the ASF legal policy
         https://www.apache.org/legal/generative-tooling.html
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] YARN-11996: Add RM JStack and Scheduler bulk-activities for diagnostic in Capacity Scheduler UI [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8765:
URL: https://github.com/apache/hadoop/pull/8765#issuecomment-5907829465

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 22s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  27m 51s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  43m 41s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 47s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shadedclient  |  14m 51s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 31s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  61m 39s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8765/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8765 |
   | Optional Tests | dupname asflicense shadedclient codespell detsecrets |
   | uname | Linux f2fb5d1c62b9 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 01cec10d6d980c36f51a928e636e1584d9fd9a75 |
   | Max. process+thread count | 630 (vs. ulimit of 10000) |
   | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8765/2/console |
   | versions | git=2.43.0 maven=3.9.15 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] YARN-11996: Add RM JStack and Scheduler bulk-activities for diagnostic in Capacity Scheduler UI [hadoop]

Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #8765:
URL: https://github.com/apache/hadoop/pull/8765#issuecomment-6034773077

   @brumi1024 Could you please take a look at this PR? Thank you very much!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] YARN-11996: Add RM JStack and Scheduler bulk-activities for diagnostic in Capacity Scheduler UI [hadoop]

Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on code in PR #8765:
URL: https://github.com/apache/hadoop/pull/8765#discussion_r4206887664


##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/components/elements/DiagnosticsDialog.tsx:
##########
@@ -99,8 +120,33 @@ export function DiagnosticsDialog() {
       description: 'Node metadata returned by /nodes.',
       data: nodes,
     },
+    {
+      id: 'bulkActivities',
+      label: 'Scheduler Bulk Activities',
+      description: 'Live response from /scheduler/bulk-activities.',
+      live: {
+        countLabel: 'Bulk activities count',
+        maxCount: MAX_DIAGNOSTIC_BULK_ACTIVITIES_COUNT,
+        fetch: (count) => apiClient.getBulkSchedulerActivities(count),
+      },
+    },
+    {
+      id: 'rmJstack',
+      label: 'ResourceManager JStack',
+      description: 'Live thread dump from /jstack (ResourceManager JVM).',
+      live: {
+        countLabel: 'ResourceManager jstack count',
+        maxCount: MAX_DIAGNOSTIC_RM_JSTACK_COUNT,
+        fetch: (count) => apiClient.getResourceManagerJstack(count),
+      },

Review Comment:
   > This PR depends on the following PR for getting the RM JStacks(by default, it is false and has to be turned on):
   
   Will these be shown if the RM JStacks are turned off?



##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/components/elements/DiagnosticsDialog.tsx:
##########
@@ -123,7 +169,9 @@ export function DiagnosticsDialog() {
 
     for (const option of datasetOptions) {
       if (selectedDatasets.includes(option.id)) {
-        (payload.datasets as Record<string, unknown>)[option.id] = option.data;
+        (payload.datasets as Record<string, unknown>)[option.id] = option.live
+          ? await option.live.fetch(Number(counts[option.id]))

Review Comment:
   getBulkActivities calls ActivitiesManager.recordAndGetBulkActivitiesInfo, which busy-waits (Thread.sleep(1) loop) until activitiesCount node-update cycles have been recorded. Is there a chance this runs past the client's default 30s timeout? If yes what happens?



##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/components/elements/DiagnosticsDialog.tsx:
##########
@@ -123,7 +169,9 @@ export function DiagnosticsDialog() {
 
     for (const option of datasetOptions) {
       if (selectedDatasets.includes(option.id)) {
-        (payload.datasets as Record<string, unknown>)[option.id] = option.data;
+        (payload.datasets as Record<string, unknown>)[option.id] = option.live

Review Comment:
   What about running these calls in parallel? Promise.all / Promise.allSettled.



##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui/src/main/webapp/src/lib/api/YarnApiClient.ts:
##########
@@ -34,9 +34,22 @@ import type {
   VersionResponse,
   YarnConfigResponse,
   ValidationResponse,
+  BulkActivitiesResponse,
 } from '~/types';
 import { HTTP_AUTH_PROPERTY, READ_ONLY_PROPERTY } from '~/config';
 
+/** Matches RMWebServices DEFAULT_ACTIVITIES_COUNT. */
+export const DEFAULT_DIAGNOSTIC_BULK_ACTIVITIES_COUNT = 10;
+
+/** Matches RMWebServices MAX_ACTIVITIES_COUNT. */
+export const MAX_DIAGNOSTIC_BULK_ACTIVITIES_COUNT = 500;
+
+/** Default jstack iterations for RM diagnostics export. */
+export const DEFAULT_DIAGNOSTIC_RM_JSTACK_COUNT = 2;
+
+/** Upper bound for RM jstack iterations in the diagnostics UI. */
+export const MAX_DIAGNOSTIC_RM_JSTACK_COUNT = 10;

Review Comment:
   Are these configurable? If yes these could be collected from the RM to avoid mismatch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Re: [PR] YARN-11996: Add RM JStack and Scheduler bulk-activities for diagnostic in Capacity Scheduler UI [hadoop]

Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8765:
URL: https://github.com/apache/hadoop/pull/8765#issuecomment-5886430663

   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   7m 20s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  detsecrets  |   0m  0s |  |  detect-secrets was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  26m 47s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  42m 35s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 46s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | +1 :green_heart: |  shadedclient  |  15m 21s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  asflicense  |   0m 31s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  67m 59s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8765/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/8765 |
   | Optional Tests | dupname asflicense shadedclient codespell detsecrets |
   | uname | Linux 21542b27782f 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 4e180224a8d7bce9871800fb97818d4b864f3275 |
   | Max. process+thread count | 609 (vs. ulimit of 10000) |
   | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8765/1/console |
   | versions | git=2.43.0 maven=3.9.15 |
   | Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org