You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Benjamin Teke (Jira)" <ji...@apache.org> on 2020/09/02 15:38:00 UTC
[jira] [Updated] (YARN-10421) Create YarnDiagnosticsServlet to
serve diagnostic queries
[ https://issues.apache.org/jira/browse/YARN-10421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Teke updated YARN-10421:
---------------------------------
Description:
YarnDiagnosticsServlet should run inside ResourceManager Daemon. The servlet forks a separate process, which executes a shell/Python/etc script. Based on the use-cases listed below the script collects information, bundles it and sends it to UI2. The diagnostic cases are the following:
# Application hanging:
** Application logs
** Find the hanging container and get multiple Jstacks
** ResourceManager logs during job lifecycle
** NodeManager logs from NodeManager where the hanging containers of the jobs ran
** Job configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL
# Application failed:
** Application logs
** ResourceManager logs during job lifecycle.
** NodeManager logs from NodeManager where the hanging containers of the jobs ran
** Job Configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL.
** Job related metrics like container, attempts.
# Scheduler related issue:
** ResourceManager Scheduler logs with DEBUG enabled for 2 minutes.
** Multiple Jstacks of ResourceManager
** YARN and Scheduler Configuration
** Cluster Scheduler API _/ws/v1/cluster/scheduler_ and Cluster Nodes API _/ws/v1/cluster/nodes response_
** Scheduler Activities _/ws/v1/cluster/scheduler/bulkactivities_ response (YARN-10319)
# ResourceManager / NodeManager daemon fails to start:
** ResourceManager and NodeManager out and log file
** YARN and Scheduler Configuration
To ease the load on the RM, the servlet should allow only one HTTP request at a time. If a new request comes in while serving another an appropriate response code should be returned, with the message "Diagnostics Collection in Progress”.
was:
YarnDiagnosticsServlet should run inside ResourceManager Daemon. The servlet forks a separate process, which executes a shell/Python/etc script. Based on the use-cases listed below the script collects information, bundles it and sends it to UI2. The diagnostic cases are the following:
# Application hanging:
** Application logs
** Find the hanging container and get multiple Jstacks
** ResourceManager logs during job lifecycle
** NodeManager logs from NodeManager where the hanging containers of the jobs ran
** Job configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL
# Application failed:
** Application logs
** ResourceManager logs during job lifecycle.
** NodeManager logs from NodeManager where the hanging containers of the jobs ran
** Job Configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL.
** Job related metrics like container, attempts.
# Scheduler related issue:
** ResourceManager Scheduler logs with DEBUG enabled for 2 minutes.
** Multiple Jstacks of ResourceManager
** YARN and Scheduler Configuration
** Cluster Scheduler API _/ws/v1/cluster/scheduler_ and Cluster Nodes API _/ws/v1/cluster/nodes response_
** Scheduler Activities _/ws/v1/cluster/scheduler/bulkactivities_ response (YARN-10319)
# ResourceManager / NodeManager daemon fails to start:
** ResourceManager and NodeManager out and log file
** YARN and Scheduler Configuration
To ease the load on the RM, the servlet should allow only one HTTP request at a time. If a new request comes in while serving another an appropriate response code should be returned, with the message "Diagnostics Collection in Progress”.
> Create YarnDiagnosticsServlet to serve diagnostic queries
> ----------------------------------------------------------
>
> Key: YARN-10421
> URL: https://issues.apache.org/jira/browse/YARN-10421
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
>
> YarnDiagnosticsServlet should run inside ResourceManager Daemon. The servlet forks a separate process, which executes a shell/Python/etc script. Based on the use-cases listed below the script collects information, bundles it and sends it to UI2. The diagnostic cases are the following:
> # Application hanging:
> ** Application logs
> ** Find the hanging container and get multiple Jstacks
> ** ResourceManager logs during job lifecycle
> ** NodeManager logs from NodeManager where the hanging containers of the jobs ran
> ** Job configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL
> # Application failed:
> ** Application logs
> ** ResourceManager logs during job lifecycle.
> ** NodeManager logs from NodeManager where the hanging containers of the jobs ran
> ** Job Configuration from MapReduce HistoryServer, Spark HistoryServer, Tez History URL.
> ** Job related metrics like container, attempts.
> # Scheduler related issue:
> ** ResourceManager Scheduler logs with DEBUG enabled for 2 minutes.
> ** Multiple Jstacks of ResourceManager
> ** YARN and Scheduler Configuration
> ** Cluster Scheduler API _/ws/v1/cluster/scheduler_ and Cluster Nodes API _/ws/v1/cluster/nodes response_
> ** Scheduler Activities _/ws/v1/cluster/scheduler/bulkactivities_ response (YARN-10319)
> # ResourceManager / NodeManager daemon fails to start:
> ** ResourceManager and NodeManager out and log file
> ** YARN and Scheduler Configuration
> To ease the load on the RM, the servlet should allow only one HTTP request at a time. If a new request comes in while serving another an appropriate response code should be returned, with the message "Diagnostics Collection in Progress”.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org