You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rares Vernica <rv...@gmail.com> on 2010/08/12 19:38:06 UTC
DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Hello,
>From time to time I get the following error:
Error initializing attempt_201008101445_0212_r_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_201008101445_0212/job.xml
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
If I restart the job without making any changes to the cluster or the
disks it eventually works. After a while I get this error again for a
different job. Restarting the job always seems to work but this is
very annoying.
I searched online and it seems that this error is triggered when there
is not enough space on any of the disks. This is not the case for me
as each node has 200GB of free space.
Is there anything else I can check besides the free space on the disks?
Thanks!
Rares
Re: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Posted by patek tek <pa...@gmail.com>.
Trying again
On Thu, Aug 12, 2010 at 1:45 PM, David Rosenstrauch <da...@darose.net>wrote:
> On 08/12/2010 01:42 PM, Rares Vernica wrote:
>
>> I forgot to mention that in my cluster the HDFS replication is set to
>> 1. I know this is not recommended but I only have 5 nodes in the
>> cluster, there are no failures
>>
>
> There will be! :-)
>
> DR
>
Re: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Posted by Rares Vernica <rv...@gmail.com>.
On Thu, Aug 12, 2010 at 10:45 AM, David Rosenstrauch <da...@darose.net> wrote:
>
> On 08/12/2010 01:42 PM, Rares Vernica wrote:
>>
>> I forgot to mention that in my cluster the HDFS replication is set to
>> 1. I know this is not recommended but I only have 5 nodes in the
>> cluster, there are no failures
>
> There will be! :-)
For the size of the cluster I have and the time-frame I am using it, I
am pretty sure there are no hardware failures right now.
It is true that I see the software failing and that restarting the job
fixes the issues, but I was not expecting the software to fail if the
hardware is OK.
Cheers,
Rares
Re: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Posted by David Rosenstrauch <da...@darose.net>.
On 08/12/2010 01:42 PM, Rares Vernica wrote:
> I forgot to mention that in my cluster the HDFS replication is set to
> 1. I know this is not recommended but I only have 5 nodes in the
> cluster, there are no failures
There will be! :-)
DR
Re: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Posted by Rares Vernica <rv...@gmail.com>.
On Thu, Aug 12, 2010 at 10:38 AM, Rares Vernica <rv...@gmail.com> wrote:
>
> Hello,
>
> From time to time I get the following error:
>
> Error initializing attempt_201008101445_0212_r_000002_0:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> any valid local directory for
> taskTracker/jobcache/job_201008101445_0212/job.xml
> at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
> at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
> at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
>
> If I restart the job without making any changes to the cluster or the
> disks it eventually works. After a while I get this error again for a
> different job. Restarting the job always seems to work but this is
> very annoying.
>
> I searched online and it seems that this error is triggered when there
> is not enough space on any of the disks. This is not the case for me
> as each node has 200GB of free space.
I forgot to mention that in my cluster the HDFS replication is set to
1. I know this is not recommended but I only have 5 nodes in the
cluster, there are no failures, and I focus on performance.
> Is there anything else I can check besides the free space on the disks?
>
> Thanks!
> Rares
Re: DiskChecker$DiskErrorException: Could not find any valid local directory -- lots of free disk space
Posted by Raj V <ra...@yahoo.com>.
I have seen this error from time to time and it has been either due to space or
missing directories or disk errors.
Space issue was caused by the fact that the I had mounted /de/sdc on /hadoop-dsk
and the mount had failed. And in another case I had
accidentally deleted hadoop.tmp.dir in a node and whenever the reduce job was
scheduled on that node that attempt would fail.
I usually look for where attempt_id ( in your case
attempt_201008101445_0212_r_000002_0) scheduled in the jobtracker , look in
that node and see if space and directories are Ok and sufficient and check for
disk errors.
Raj
________________________________
From: Rares Vernica <rv...@gmail.com>
To: common-user@hadoop.apache.org
Sent: Thu, August 12, 2010 10:38:06 AM
Subject: DiskChecker$DiskErrorException: Could not find any valid local
directory -- lots of free disk space
Hello,
>From time to time I get the following error:
Error initializing attempt_201008101445_0212_r_000002_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_201008101445_0212/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1664)
at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:97)
at
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1629)
If I restart the job without making any changes to the cluster or the
disks it eventually works. After a while I get this error again for a
different job. Restarting the job always seems to work but this is
very annoying.
I searched online and it seems that this error is triggered when there
is not enough space on any of the disks. This is not the case for me
as each node has 200GB of free space.
Is there anything else I can check besides the free space on the disks?
Thanks!
Rares