You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Lavkesh Lahngir <la...@linux.com> on 2015/05/06 16:23:34 UTC

Resource localisation on failed disk causes containers to fail

Hello,

If a resource is localised on a disk and that disk has gone bad after
localising, subsequent containers are not able to find the resource and NM
does not download it again.
The problem is stat system call succeeds on the bad path which causes
file.exists() to return true.
But ls on the path returns an IO error.

LocalResourcesTrackerImpl.java
case REQUEST:
if (rsrc != null && (!isResourcePresent(rsrc))) {
    LOG.info("Resource " + rsrc.getLocalPath()
            + " is missing, localizing it again");
    removeResource(req);
    rsrc = null;
}
if (null == rsrc) {
    rsrc = new LocalizedResource(req, dispatcher);
    localrsrc.put(req, rsrc);
}
break;

isResourcePresent() calls file.exists() which calls stat64 natively which
returns true.. But the disk actually is bad, and there is no possibility of
reading/writing on that path.

example:
>>stat /data/d3/yarn/local

  File: `/data/d3/yarn/local'
Size: 4096      Blocks: 8          IO Block: 4096   directory
Device: 830h/2096d Inode: 107307009   Links: 3
Access: (0755/drwxr-xr-x)  Uid: (  110/ yarn)   Gid: (  118/  hadoop)
Access: 2014-11-18 13:57:19.000000000 +0000
Modify: 2014-11-19 11:15:15.000000000 +0000
Change: 2014-11-19 11:15:15.000000000 +0000
Birth: -

and ls  says

ls: reading directory /data/d3/mapred: Input/output error


Any thoughts?

Thanks

Re: Resource localisation on failed disk causes containers to fail

Posted by Lavkesh Lahngir <la...@linux.com>.
correction:
>>ls  /data/d3/yarn/local
ls: reading directory /data/d3/yarn/local: Input/output error

Thanks

On Wed, May 6, 2015 at 7:53 PM, Lavkesh Lahngir <la...@linux.com> wrote:

> Hello,
>
> If a resource is localised on a disk and that disk has gone bad after
> localising, subsequent containers are not able to find the resource and NM
> does not download it again.
> The problem is stat system call succeeds on the bad path which causes
> file.exists() to return true.
> But ls on the path returns an IO error.
>
> LocalResourcesTrackerImpl.java
> case REQUEST:
> if (rsrc != null && (!isResourcePresent(rsrc))) {
>     LOG.info("Resource " + rsrc.getLocalPath()
>             + " is missing, localizing it again");
>     removeResource(req);
>     rsrc = null;
> }
> if (null == rsrc) {
>     rsrc = new LocalizedResource(req, dispatcher);
>     localrsrc.put(req, rsrc);
> }
> break;
>
> isResourcePresent() calls file.exists() which calls stat64 natively which
> returns true.. But the disk actually is bad, and there is no possibility of
> reading/writing on that path.
>
> example:
> >>stat /data/d3/yarn/local
>
>   File: `/data/d3/yarn/local'
> Size: 4096      Blocks: 8          IO Block: 4096   directory
> Device: 830h/2096d Inode: 107307009   Links: 3
> Access: (0755/drwxr-xr-x)  Uid: (  110/ yarn)   Gid: (  118/  hadoop)
> Access: 2014-11-18 13:57:19.000000000 +0000
> Modify: 2014-11-19 11:15:15.000000000 +0000
> Change: 2014-11-19 11:15:15.000000000 +0000
> Birth: -
>
> and ls  says
>
> ls: reading directory /data/d3/mapred: Input/output error
>
>
> Any thoughts?
>
> Thanks
>