[jira] [Commented] (MAPREDUCE-3121) NodeManager should handle disk-failures

["Ravi Gummadi (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13143873#comment-13143873 ] 

Ravi Gummadi commented on MAPREDUCE-3121:
-----------------------------------------

Code level summary of the patch attached:

(1) Node Manager launches DiskHealthCheckeraService that periodically launches/executes the disk-health-check-code. A new configuration property yarn.nodemanger.disk-health-checker.interval-ms is added for the frequency with which this code is executed, with a default value of 120*1000ms(i.e. 2minutes).

(2) LocalStorage is a new class that manages a list of local file system directories and provides api for checking the health of those directories (mostly similar to TaskTracker.LocalStorgae class of 0.20 except that this class' checkDirs() doesn't throw DiskErrorException when all directories fail but returns true if a new disk-failure is seen.

(3) DiskHealthCheckerService maintains 2 LocalStorage objects ---- one for nm-local-dirs and second one for nm-log-dirs.

(4) ContainerExecutor is initialized with the DiskHealthCheckerService object. So both DefaultContainerExecutor.java and LinuxContainerExecutor.java get good nm-local-dirs and nm-log-dirs from the DiskHealthChecker always.

(5) container-executor binary gets good nm-local-dirs and good nm-log-dirs as a parameter and uses these good dirs only. So these are removed from the configuration file(i.e. not needed to be configured in container-executor.cfg configuration file).

(6) Whenever a new container gets launched, the good nm-local-dirs and good nm-log-dirs are updated in the configuration sothat containers won't access bad disks. Everybody (localizer, webserver) goes through DiskHealthChecker to access nm-local-dirs and nm-log-dirs.

(7) On the NodeManager web UI, NodeHealthReport will be showing the lost of good nm-local-dirs and nm-log-dirs in addition to true/false about the health of the node.

(8) A new unit test TestDiskFailures is added that makes disks(both nm-local-dirs and nm-log-dirs) fail and checks/validates if the NodeManager/DiskHealthChecker can identifies these disk-failures or not.

\\
Tested the patch with (1) DefaultContainerExecutor and (2) LinuxContainerExecutor on my single node cluster. The functionality seems to be working fine with disk failures getting identified by NodeManager and the bad nm-local-dirs and bad nm-log-dirs getting avoided for new containers.
                
> NodeManager should handle disk-failures
> ---------------------------------------
>
>                 Key: MAPREDUCE-3121
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, nodemanager
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Ravi Gummadi
>             Fix For: 0.23.1
>
>         Attachments: 3121.patch
>
>
> This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira