[jira] [Commented] (YARN-3591) Resource Localisation on a bad disk causes subsequent containers failure

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727155#comment-14727155 ] 

Hadoop QA commented on YARN-3591:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  19m 33s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new or modified test files. |
| {color:green}+1{color} | javac |  10m 43s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |  12m 13s | There were no new javadoc warning messages. |
| {color:green}+1{color} | release audit |   0m 25s | The applied patch does not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 48s | The applied patch generated  2 new checkstyle issues (total was 172, now 174). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that end in whitespace. |
| {color:green}+1{color} | install |   1m 54s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 41s | The patch built with eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   1m 29s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests |   8m 10s | Tests passed in hadoop-yarn-server-nodemanager. |
| | |  56m  0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12753729/YARN-3591.8.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 095ab9a |
| checkstyle |  https://builds.apache.org/job/PreCommit-YARN-Build/8970/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt |
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8970/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8970/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8970/console |


This message was automatically generated.

> Resource Localisation on a bad disk causes subsequent containers failure 
> -------------------------------------------------------------------------
>
>                 Key: YARN-3591
>                 URL: https://issues.apache.org/jira/browse/YARN-3591
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.7.0
>            Reporter: Lavkesh Lahngir
>            Assignee: Lavkesh Lahngir
>         Attachments: 0001-YARN-3591.1.patch, 0001-YARN-3591.patch, YARN-3591.2.patch, YARN-3591.3.patch, YARN-3591.4.patch, YARN-3591.5.patch, YARN-3591.6.patch, YARN-3591.7.patch, YARN-3591.8.patch
>
>
> It happens when a resource is localised on the disk, after localising that disk has gone bad. NM keeps paths for localised resources in memory.  At the time of resource request isResourcePresent(rsrc) will be called which calls file.exists() on the localised path.
> In some cases when disk has gone bad, inodes are stilled cached and file.exists() returns true. But at the time of reading, file will not open.
> Note: file.exists() actually calls stat64 natively which returns true because it was able to find inode information from the OS.
> A proposal is to call file.list() on the parent path of the resource, which will call open() natively. If the disk is good it should return an array of paths with length at-least 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)