You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hean-Chhinling (via GitHub)" <gi...@apache.org> on 2026/09/16 10:38:43 UTC
[PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Hean-Chhinling opened a new pull request, #8742:
URL: https://github.com/apache/hadoop/pull/8742
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Upon agree, the application_failed & application hanging combines into "Application Diagnostic". This endpoint will be called by YARN UI2.
The scheduler_related_issue endpoint will be called by the new YARN QueueManager UI for capacity scheduler ([PR](https://github.com/brumi1024/yarn-scheduler-ui/pull/180/changes)).
The RM_NM_startup failures have been removed as it is not feasible to implement a UI for it in real case scenario.
### How was this patch tested?
Manually, and will write a unit-test for it.
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? *Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4103133136
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
+ app_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/apps/{}/jstack/{}".format(app_id, _NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "application_jstack", app_jstack)
+
+ nm_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "nm_{}_jstack".format(nm_address), nm_jstack)
+
+ app_info = _create_request(_rm_url("ws/v1/cluster/apps/{}".format(app_id)))
+ _write_output(output_path, "application_info", app_info)
+
+ app_attempts = _create_request(_rm_url("ws/v1/cluster/apps/{}/appattempts".format(app_id)))
+ _write_output(output_path, "application_attempts", app_attempts)
+
+ start_time, end_time = _get_application_time(app_info)
+
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "resourcemanager_log",
+ _filter_node_log(log_address, start_time, end_time, _NODE_SCHEME))
+
+ if "amHostHttpAddress" in app_info:
+ app_info_xml = ET.fromstring(app_info)
+ nm_address = app_info_xml.find("amHostHttpAddress").text
+ log_address = _get_node_log_address(nm_address, _NM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "nodemanager_log",
+ _get_container_log(log_address, app_id, _NODE_SCHEME))
+
+ command = _run_cmd_and_save_output(os.path.join(output_path, "app_logs"), app_id,
+ "yarn", "logs", "-applicationId", app_id)
+ command.communicate()
+ logger.info("Application diagnostics written to %s", output_path)
+ return output_path
+
+
+def scheduler_related_issue() -> str:
+ """
+ ResourceManager Scheduler Logs with DEBUG enabled for 2 minutes.
+ Multiple Jstack of ResourceManager
+ YARN and Scheduler Configuration
+ Cluster Scheduler API /ws/v1/cluster/scheduler and Cluster Nodes API /ws/v1/cluster/nodes response
+ Scheduler Activities /ws/v1/cluster/scheduler/bulk-activities response
+ """
+ logger.info("Collecting scheduler-related diagnostics")
+ output_path = _create_output_dir(
+ os.path.join(_TEMP_DIR, "scheduler_related_issue" + str(time.time()).split(".")[0]))
+
+ rm_jstack = _create_request(_rm_url("ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)), False)
+ _write_output(output_path, "rm_{}_jstack".format(_RM_ADDRESS), rm_jstack)
+
+ scheduler_info = _create_request(_rm_url("ws/v1/cluster/scheduler"))
+ _write_output(output_path, "scheduler_info", scheduler_info)
+
+ nodes_info = _create_request(_rm_url("ws/v1/cluster/nodes"))
+ _write_output(output_path, "nodemanager_info", nodes_info)
+
+ scheduler_activities = _create_request(_rm_url("ws/v1/cluster/scheduler/bulk-activities"))
+ _write_output(output_path, "scheduler_activities", scheduler_activities)
+
+ scheduler_config = _create_request(_rm_url("ws/v1/cluster/scheduler-conf"))
+ _write_output(output_path, "scheduler_configuration", scheduler_config)
+
+ yarn_conf = _run_command("cat", os.path.join(_HADOOP_CONF_DIR, _YARN_SITE_XML))
+ _write_output(output_path, "yarn_site", yarn_conf)
+
+ enable_debug_log = _set_rm_scheduler_log_level("DEBUG")
+ logger.info("Set RM scheduler log level to DEBUG: %s", enable_debug_log)
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ start_time, end_time = (_format_datetime_no_seconds(datetime.now() - timedelta(seconds=120)),
Review Comment:
Nice catch, I haven't thought this in detail.
I will implement that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4103028465
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
+ app_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/apps/{}/jstack/{}".format(app_id, _NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "application_jstack", app_jstack)
+
+ nm_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "nm_{}_jstack".format(nm_address), nm_jstack)
+
+ app_info = _create_request(_rm_url("ws/v1/cluster/apps/{}".format(app_id)))
+ _write_output(output_path, "application_info", app_info)
+
+ app_attempts = _create_request(_rm_url("ws/v1/cluster/apps/{}/appattempts".format(app_id)))
+ _write_output(output_path, "application_attempts", app_attempts)
+
+ start_time, end_time = _get_application_time(app_info)
+
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "resourcemanager_log",
+ _filter_node_log(log_address, start_time, end_time, _NODE_SCHEME))
+
+ if "amHostHttpAddress" in app_info:
+ app_info_xml = ET.fromstring(app_info)
+ nm_address = app_info_xml.find("amHostHttpAddress").text
+ log_address = _get_node_log_address(nm_address, _NM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "nodemanager_log",
+ _get_container_log(log_address, app_id, _NODE_SCHEME))
+
+ command = _run_cmd_and_save_output(os.path.join(output_path, "app_logs"), app_id,
+ "yarn", "logs", "-applicationId", app_id)
+ command.communicate()
+ logger.info("Application diagnostics written to %s", output_path)
+ return output_path
+
+
+def scheduler_related_issue() -> str:
+ """
+ ResourceManager Scheduler Logs with DEBUG enabled for 2 minutes.
+ Multiple Jstack of ResourceManager
+ YARN and Scheduler Configuration
+ Cluster Scheduler API /ws/v1/cluster/scheduler and Cluster Nodes API /ws/v1/cluster/nodes response
+ Scheduler Activities /ws/v1/cluster/scheduler/bulk-activities response
+ """
+ logger.info("Collecting scheduler-related diagnostics")
+ output_path = _create_output_dir(
+ os.path.join(_TEMP_DIR, "scheduler_related_issue" + str(time.time()).split(".")[0]))
+
+ rm_jstack = _create_request(_rm_url("ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)), False)
Review Comment:
Jstack is part of the design document:
<img width="1277" height="650" alt="Screenshot 2026-09-25 at 15 44 30" src="https://github.com/user-attachments/assets/86c81ab7-7f4e-4aca-bd8c-a5d9fb20e8a3" />
There is no easy way to collect Jstack nor is there a trivial way to do.
Therefore, I created 3 new endpoints to collect Jstack for Application containers, NodeManager and ResourceManager.
This PR is depend on this PR:
https://github.com/apache/hadoop/pull/8123
I will correct the endpoint here from `ws/v1/node/jstack` -> `ws/v1/cluster/jstack` since this is the right one for RM.
>Additionally, curl does not currently fail on HTTP error responses, so a 404
response could be saved as a successful *_jstack result.
Good idea, at this moment it will just write the 404 to the file. But at the same time, this a better than an error in the console log because the user would just download the necessary files instead of viewing the console.
But please let me know what do you think :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4103133136
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
+ app_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/apps/{}/jstack/{}".format(app_id, _NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "application_jstack", app_jstack)
+
+ nm_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "nm_{}_jstack".format(nm_address), nm_jstack)
+
+ app_info = _create_request(_rm_url("ws/v1/cluster/apps/{}".format(app_id)))
+ _write_output(output_path, "application_info", app_info)
+
+ app_attempts = _create_request(_rm_url("ws/v1/cluster/apps/{}/appattempts".format(app_id)))
+ _write_output(output_path, "application_attempts", app_attempts)
+
+ start_time, end_time = _get_application_time(app_info)
+
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "resourcemanager_log",
+ _filter_node_log(log_address, start_time, end_time, _NODE_SCHEME))
+
+ if "amHostHttpAddress" in app_info:
+ app_info_xml = ET.fromstring(app_info)
+ nm_address = app_info_xml.find("amHostHttpAddress").text
+ log_address = _get_node_log_address(nm_address, _NM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "nodemanager_log",
+ _get_container_log(log_address, app_id, _NODE_SCHEME))
+
+ command = _run_cmd_and_save_output(os.path.join(output_path, "app_logs"), app_id,
+ "yarn", "logs", "-applicationId", app_id)
+ command.communicate()
+ logger.info("Application diagnostics written to %s", output_path)
+ return output_path
+
+
+def scheduler_related_issue() -> str:
+ """
+ ResourceManager Scheduler Logs with DEBUG enabled for 2 minutes.
+ Multiple Jstack of ResourceManager
+ YARN and Scheduler Configuration
+ Cluster Scheduler API /ws/v1/cluster/scheduler and Cluster Nodes API /ws/v1/cluster/nodes response
+ Scheduler Activities /ws/v1/cluster/scheduler/bulk-activities response
+ """
+ logger.info("Collecting scheduler-related diagnostics")
+ output_path = _create_output_dir(
+ os.path.join(_TEMP_DIR, "scheduler_related_issue" + str(time.time()).split(".")[0]))
+
+ rm_jstack = _create_request(_rm_url("ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)), False)
+ _write_output(output_path, "rm_{}_jstack".format(_RM_ADDRESS), rm_jstack)
+
+ scheduler_info = _create_request(_rm_url("ws/v1/cluster/scheduler"))
+ _write_output(output_path, "scheduler_info", scheduler_info)
+
+ nodes_info = _create_request(_rm_url("ws/v1/cluster/nodes"))
+ _write_output(output_path, "nodemanager_info", nodes_info)
+
+ scheduler_activities = _create_request(_rm_url("ws/v1/cluster/scheduler/bulk-activities"))
+ _write_output(output_path, "scheduler_activities", scheduler_activities)
+
+ scheduler_config = _create_request(_rm_url("ws/v1/cluster/scheduler-conf"))
+ _write_output(output_path, "scheduler_configuration", scheduler_config)
+
+ yarn_conf = _run_command("cat", os.path.join(_HADOOP_CONF_DIR, _YARN_SITE_XML))
+ _write_output(output_path, "yarn_site", yarn_conf)
+
+ enable_debug_log = _set_rm_scheduler_log_level("DEBUG")
+ logger.info("Set RM scheduler log level to DEBUG: %s", enable_debug_log)
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ start_time, end_time = (_format_datetime_no_seconds(datetime.now() - timedelta(seconds=120)),
Review Comment:
Nice catch, I haven't thought this in detail.
I will implement to that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#issuecomment-5834243010
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 14m 26s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | mvnsite | 1m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | shadedclient | 30m 46s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 31m 18s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| -1 :x: | pylint | 0m 4s | [/results-pylint.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8742/3/artifact/out/results-pylint.txt) | The patch generated 32 new + 0 unchanged - 0 fixed = 32 total (was 0) |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| -1 :x: | shadedclient | 29m 38s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 116m 18s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8742/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 252m 47s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8742/3/artifact/out/Dockerfile |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets pylint |
| uname | Linux 9a2f7c5955f0 5.15.0-186-generic #196-Ubuntu SMP Sat Jun 20 16:09:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 90f0d1da37c294cf1bb9b379ce6af9111ee20f4f |
| Default Java | Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8742/3/testReport/ |
| Max. process+thread count | 982 (vs. ulimit of 10000) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8742/3/console |
| versions | git=2.43.0 maven=3.9.15 pylint=3.0.3 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "Hean-Chhinling (via GitHub)" <gi...@apache.org>.
Hean-Chhinling commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4103143395
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
Review Comment:
Very good idea, I have not really thought about this case.
I will do that in my next commit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4092982313
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
Review Comment:
Could we handle applications that do not yet have an AM container?
For example, an application stuck in ACCEPTED while waiting for resources may not have `amHostHttpAddress`. `_get_nodemanager_address()` unconditionally accesses `.find("amHostHttpAddress").text`, so this case raises AttributeError before application information, attempts, or RM logs are collected.
This is an important case for application-hang diagnostics.
Please save the application information first, skip NodeManager-specific collection when the
address is unavailable, and continue collecting the remaining diagnostics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4092958951
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
+ app_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/apps/{}/jstack/{}".format(app_id, _NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "application_jstack", app_jstack)
+
+ nm_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "nm_{}_jstack".format(nm_address), nm_jstack)
+
+ app_info = _create_request(_rm_url("ws/v1/cluster/apps/{}".format(app_id)))
+ _write_output(output_path, "application_info", app_info)
+
+ app_attempts = _create_request(_rm_url("ws/v1/cluster/apps/{}/appattempts".format(app_id)))
+ _write_output(output_path, "application_attempts", app_attempts)
+
+ start_time, end_time = _get_application_time(app_info)
+
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "resourcemanager_log",
+ _filter_node_log(log_address, start_time, end_time, _NODE_SCHEME))
+
+ if "amHostHttpAddress" in app_info:
+ app_info_xml = ET.fromstring(app_info)
+ nm_address = app_info_xml.find("amHostHttpAddress").text
+ log_address = _get_node_log_address(nm_address, _NM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "nodemanager_log",
+ _get_container_log(log_address, app_id, _NODE_SCHEME))
+
+ command = _run_cmd_and_save_output(os.path.join(output_path, "app_logs"), app_id,
+ "yarn", "logs", "-applicationId", app_id)
+ command.communicate()
+ logger.info("Application diagnostics written to %s", output_path)
+ return output_path
+
+
+def scheduler_related_issue() -> str:
+ """
+ ResourceManager Scheduler Logs with DEBUG enabled for 2 minutes.
+ Multiple Jstack of ResourceManager
+ YARN and Scheduler Configuration
+ Cluster Scheduler API /ws/v1/cluster/scheduler and Cluster Nodes API /ws/v1/cluster/nodes response
+ Scheduler Activities /ws/v1/cluster/scheduler/bulk-activities response
+ """
+ logger.info("Collecting scheduler-related diagnostics")
+ output_path = _create_output_dir(
+ os.path.join(_TEMP_DIR, "scheduler_related_issue" + str(time.time()).split(".")[0]))
+
+ rm_jstack = _create_request(_rm_url("ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)), False)
+ _write_output(output_path, "rm_{}_jstack".format(_RM_ADDRESS), rm_jstack)
+
+ scheduler_info = _create_request(_rm_url("ws/v1/cluster/scheduler"))
+ _write_output(output_path, "scheduler_info", scheduler_info)
+
+ nodes_info = _create_request(_rm_url("ws/v1/cluster/nodes"))
+ _write_output(output_path, "nodemanager_info", nodes_info)
+
+ scheduler_activities = _create_request(_rm_url("ws/v1/cluster/scheduler/bulk-activities"))
+ _write_output(output_path, "scheduler_activities", scheduler_activities)
+
+ scheduler_config = _create_request(_rm_url("ws/v1/cluster/scheduler-conf"))
+ _write_output(output_path, "scheduler_configuration", scheduler_config)
+
+ yarn_conf = _run_command("cat", os.path.join(_HADOOP_CONF_DIR, _YARN_SITE_XML))
+ _write_output(output_path, "yarn_site", yarn_conf)
+
+ enable_debug_log = _set_rm_scheduler_log_level("DEBUG")
+ logger.info("Set RM scheduler log level to DEBUG: %s", enable_debug_log)
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ start_time, end_time = (_format_datetime_no_seconds(datetime.now() - timedelta(seconds=120)),
Review Comment:
This collects the previous two minutes immediately after enabling DEBUG, rather than collecting two minutes of DEBUG output.
Unless DEBUG was already enabled, most of this interval will not contain the scheduler diagnostics that this operation intends to capture.
Could we record the start time when DEBUG is enabled, wait for the collection interval, and then retrieve that interval's logs? Please also restore the previous log level in a `finally` block instead of unconditionally setting INFO, so failures do not leave DEBUG enabled or overwrite an existing setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-10422: add python script for collecting diagnostic data [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on code in PR #8742:
URL: https://github.com/apache/hadoop/pull/8742#discussion_r4092948970
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/diagnostics/diagnostics_collector.py:
##########
@@ -0,0 +1,358 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import logging
+import os
+import re
+import socket
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from datetime import datetime, timedelta
+from typing import List, Optional, Tuple
+
+logger = logging.getLogger(__name__)
+
+_TEMP_DIR = "/tmp"
+_HADOOP_CONF_DIR: Optional[str] = None
+_YARN_SITE_XML = "yarn-site.xml"
+_RM_WEBAPP_HTTPS_ADDRESS_KEY = "yarn.resourcemanager.webapp.https.address"
+_RM_WEBAPP_HTTP_ADDRESS_KEY = "yarn.resourcemanager.webapp.address"
+_RM_ADDRESS: Optional[str] = None
+_NODE_SCHEME = "http"
+
+_RM_LOG_REGEX = r"(?<=\")\/logs.+?RESOURCEMANAGER.+?(?=\")"
+_NM_LOG_REGEX = r"(?<=\")\/logs.+?NODEMANAGER.+?(?=\")"
+_INPUT_TIME_FORMAT = '%a %b %d %H:%M:%S %Z %Y' # e.g. Wed May 28 07:35:39 UTC 2025
+_OUTPUT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S,%f' # e.g. 2025-05-28 11:57:05,435
+_OUTPUT_TIME_FORMAT_WITHOUT_SECOND = '%Y-%m-%d %H:%M' # e.g. 2025-05-28 11:57
+_NUMBER_OF_JSTACK = 3
+
+args = None
+
+
+def list_issues() -> None:
+ print("application_diagnostic:appId", "scheduler_related_issue", sep="\n")
+
+
+def application_diagnostic() -> str:
+ """
+ Application Logs, Application Info, Application Attempts
+ Multiple JStack of Hanging Containers and NodeManager
+ ResourceManager logs during job duration.
+ NodeManager logs from NodeManager where hanging containers of jobs run during the duration of containers.
+ """
+ if args.arguments is None or len(args.arguments) == 0:
+ logger.error("Missing application or job id")
+ sys.exit(os.EX_USAGE)
+
+ app_id = args.arguments[0]
+ logger.info("Collecting application diagnostics for %s", app_id)
+ output_path = _create_output_dir(os.path.join(_TEMP_DIR, app_id))
+
+ nm_address = _get_nodemanager_address(app_id)
+ app_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/apps/{}/jstack/{}".format(app_id, _NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "application_jstack", app_jstack)
+
+ nm_jstack = _create_request(
+ _web_url(_NODE_SCHEME, nm_address, "ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)),
+ False)
+ _write_output(output_path, "nm_{}_jstack".format(nm_address), nm_jstack)
+
+ app_info = _create_request(_rm_url("ws/v1/cluster/apps/{}".format(app_id)))
+ _write_output(output_path, "application_info", app_info)
+
+ app_attempts = _create_request(_rm_url("ws/v1/cluster/apps/{}/appattempts".format(app_id)))
+ _write_output(output_path, "application_attempts", app_attempts)
+
+ start_time, end_time = _get_application_time(app_info)
+
+ log_address = _get_node_log_address(_RM_ADDRESS, _RM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "resourcemanager_log",
+ _filter_node_log(log_address, start_time, end_time, _NODE_SCHEME))
+
+ if "amHostHttpAddress" in app_info:
+ app_info_xml = ET.fromstring(app_info)
+ nm_address = app_info_xml.find("amHostHttpAddress").text
+ log_address = _get_node_log_address(nm_address, _NM_LOG_REGEX, _NODE_SCHEME)
+ _write_output(os.path.join(output_path, "node_log"), "nodemanager_log",
+ _get_container_log(log_address, app_id, _NODE_SCHEME))
+
+ command = _run_cmd_and_save_output(os.path.join(output_path, "app_logs"), app_id,
+ "yarn", "logs", "-applicationId", app_id)
+ command.communicate()
+ logger.info("Application diagnostics written to %s", output_path)
+ return output_path
+
+
+def scheduler_related_issue() -> str:
+ """
+ ResourceManager Scheduler Logs with DEBUG enabled for 2 minutes.
+ Multiple Jstack of ResourceManager
+ YARN and Scheduler Configuration
+ Cluster Scheduler API /ws/v1/cluster/scheduler and Cluster Nodes API /ws/v1/cluster/nodes response
+ Scheduler Activities /ws/v1/cluster/scheduler/bulk-activities response
+ """
+ logger.info("Collecting scheduler-related diagnostics")
+ output_path = _create_output_dir(
+ os.path.join(_TEMP_DIR, "scheduler_related_issue" + str(time.time()).split(".")[0]))
+
+ rm_jstack = _create_request(_rm_url("ws/v1/node/jstack/{}".format(_NUMBER_OF_JSTACK)), False)
Review Comment:
Could we use endpoints that exist in the target Hadoop version, or explicitly identify the prerequisite changes that introduce these endpoints?
I could not find implementations for `/ws/v1/node/jstack/{count}` or `/ws/v1/node/apps/{appId}/jstack/{count}` in the PR's base revision.
The existing daemon thread-dump endpoint is `/stacks`. The same issue applies to the application and NodeManager requests above.
Additionally, curl does not currently fail on HTTP error responses, so a 404
response could be saved as a successful `*_jstack` result.
Please validate the HTTP status and report unavailable diagnostics explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org