[jira] [Commented] (YARN-10422) Create the script responsible for collecting the bundle data

["Benjamin Teke (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210928#comment-17210928 ] 

Benjamin Teke commented on YARN-10422:
--------------------------------------

Uploaded a POC patch. Currently two of the diagnostic cases are implemented, and there are some TODOs, regarding (time) filtering the logs, user permissions (for yarn logs command) and authentication. The structure of the script is completely up for debate.

> Create the script responsible for collecting the bundle data
> ------------------------------------------------------------
>
>                 Key: YARN-10422
>                 URL: https://issues.apache.org/jira/browse/YARN-10422
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Benjamin Teke
>            Assignee: Benjamin Teke
>            Priority: Major
>         Attachments: YARN-10422.POC.001.patch, YARN-10422.POC.002.patch
>
>
> The script should provide the list of diagnostic use-cases described in YARN-10421. If a request comes in to the YarnDiagnosticCollector servlet, the script will be invoked. It collects all the information required for that diagnostic category and saves it into a configurable directory as a compressed tar file. 
> An example of how the script could look like:
> {code:java}
> if [$1 = "listcommonissues"]    
>   echo "1, Application Failed"    
>   echo "2, Application Hanging"    
>   echo "3, Scheduler Related Issue"    
>   echo "4, RM failure to start"    
>   echo "5, NM failure to start"
> elif [$1 = "collect"]
>   if [$2 == 1]       
>     appId = $3      
>     mkdir /tmp/$appId
>     yarn logs -applicationId $appId > /tmp/$appId/joblogs   
>     curl <JHS>/{appId}/conf > /tmp/$appId/conf      
>     curl <RM>/logs | grep container > /tmp/$appId/rmlogs      
>     curl <NM>/logs | grep container > /tmp/$appId/nmlogs      
>     outputpath = /tmp/$appId    
>   elif      ...    
>   elif      ...    
> fi   tar and compress outputpath.{code}
>  
> During class load YarnDiagnosticsCollector reads the list of common issues from the script and keeps it in memory. On every startup of YARN UI2 diagnostics page, it fetches the list from the servlet and displays them. The servlet should handle the script changes, so if a new diagnostic case is added,  a YARN UI2 reload should show it. This way the users can easily plug new categories without any UI2 or Servlet code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org