You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rakhi Khatwani <ra...@gmail.com> on 2009/04/16 09:45:13 UTC
No space left on device Exception
Hi,
I am running a map-reduce program on 6-Node ec2 cluster. and after a
couple of hours all my tasks gets hanged.
so i started digging into the logs....
there were no logs for regionserver
no logs for tasktracker.
However for jobtracker i get the following:
2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 50002, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false, true,
10745) from 10.254.27.79:44222: error: java.io.IOException:
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
space left on device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at
org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
at java.io.BufferedWriter.close(BufferedWriter.java:248)
at java.io.PrintWriter.close(PrintWriter.java:295)
at
org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
at
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
at org.apache.hadoop.mapred.JobInProgress.comp
following are the disk information on dfs UI
domU-12-31-39-00-0C-A1<http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.8321.19391.360.2
94.672353 domU-12-31-39-00-16-F1<http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.4621.24391.670.11
94.752399 domU-12-31-39-00-45-71<http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.6421.34391.40.16
94.682303 domU-12-31-39-00-E5-D2<http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.6621.53391.180.16
94.632319 domU-12-31-39-01-64-12<http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service413.380.6421.24391.490.16
94.712264 domU-12-31-39-01-78-D1<http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.4921.24391.650.12
94.741952
I m using hadoop 0.19.0 and hbase 0.19.0
n googling the error i came arcoss the JIRA issue
http://issues.apache.org/jira/browse/HADOOP-4163
which says tht its been fixed in this version. :(
Has anyone else come up with this exception?
how do we check the maximum capacity for usable dfs and non usable dfs.
Thanks
Raakhi,
Re: No space left on device Exception
Posted by Pankil Doshi <fo...@gmail.com>.
Hey
what's your input size?
from the info you gave it seems you have used 4.2GB and so probably if thats
your input size your intermediate results mostly is less then your input.but
that too depends on your map function. Make sure about the size of
intermediate results.
Pankil
On Thu, Apr 16, 2009 at 3:25 AM, Rakhi Khatwani <ra...@gmail.com>wrote:
> Thanks,
> I will check tht
>
> Regards,
> Raakhi
>
> On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne <mi...@inf.ed.ac.uk> wrote:
>
> > it may be that intermediate results are filling your disks and when
> > the jobs crash, this all gets deleted. so it would look like you have
> > spare space when in reality you don't.
> >
> > i would check on the file system as your jobs run and see if indeed
> > they are filling-up.
> >
> > Miles
> >
> > 2009/4/16 Rakhi Khatwani <ra...@gmail.com>:
> > > Hi,
> > > following is the output on the df command
> > > [root@domU-12-31-39-00-E5-D2 conf]# df -h
> > > Filesystem Size Used Avail Use% Mounted on
> > > /dev/sda1 9.9G 4.2G 5.2G 45% /
> > > /dev/sdb 414G 924M 392G 1% /mnt
> > >
> > > from the o/p it seems that i have quite an amount of memory available.
> > but i
> > > still get the exception :(
> > >
> > > Thanks
> > > Raakhi
> > >
> > > On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.desai@hp.com
> > >wrote:
> > >
> > >> From the exception it appears that there is no space left on machine.
> > You
> > >> can check using 'df'
> > >>
> > >> Thanks
> > >> Milind
> > >>
> > >> -----Original Message-----
> > >> From: Rakhi Khatwani [mailto:rakhi.khatwani@gmail.com]
> > >> Sent: Thursday, April 16, 2009 1:15 PM
> > >> To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
> > >> Subject: No space left on device Exception
> > >>
> > >> Hi,
> > >> I am running a map-reduce program on 6-Node ec2 cluster. and after
> a
> > >> couple of hours all my tasks gets hanged.
> > >>
> > >> so i started digging into the logs....
> > >>
> > >> there were no logs for regionserver
> > >> no logs for tasktracker.
> > >> However for jobtracker i get the following:
> > >>
> > >> 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> > >> handler 9 on 50002, call
> > >> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
> > >> true,
> > >> 10745) from 10.254.27.79:44222: error: java.io.IOException:
> > >> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> > device
> > >> java.io.IOException: org.apache.hadoop.fs.FSError:
> java.io.IOException:
> > No
> > >> space left on device
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> > >> at
> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> > >> at
> > java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> > >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> > >> at
> > >> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
> > >> at
> > org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> > >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> > >> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> > >> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
> > >> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
> > >> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
> > >> at java.io.BufferedWriter.close(BufferedWriter.java:248)
> > >> at java.io.PrintWriter.close(PrintWriter.java:295)
> > >> at
> > >>
> > >>
> >
> org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
> > >> at
> > >>
> >
> org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
> > >> at org.apache.hadoop.mapred.JobInProgress.comp
> > >>
> > >>
> > >>
> > >> following are the disk information on dfs UI
> > >> domU-12-31-39-00-0C-A1<
> > >>
> >
> http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >0In
> > >> Service413.380.8321.19391.360.2
> > >> 94.672353 domU-12-31-39-00-16-F1<
> > >>
> >
> http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >1In
> > >> Service413.380.4621.24391.670.11
> > >> 94.752399 domU-12-31-39-00-45-71<
> > >>
> >
> http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >1In
> > >> Service413.380.6421.34391.40.16
> > >> 94.682303 domU-12-31-39-00-E5-D2<
> > >>
> >
> http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >0In
> > >> Service413.380.6621.53391.180.16
> > >> 94.632319 domU-12-31-39-01-64-12<
> > >>
> >
> http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >2In
> > >> Service413.380.6421.24391.490.16
> > >> 94.712264 domU-12-31-39-01-78-D1<
> > >>
> >
> http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> > >> >0In
> > >> Service413.380.4921.24391.650.12
> > >> 94.741952
> > >>
> > >> I m using hadoop 0.19.0 and hbase 0.19.0
> > >>
> > >> n googling the error i came arcoss the JIRA issue
> > >> http://issues.apache.org/jira/browse/HADOOP-4163
> > >>
> > >> which says tht its been fixed in this version. :(
> > >>
> > >> Has anyone else come up with this exception?
> > >>
> > >> how do we check the maximum capacity for usable dfs and non usable
> dfs.
> > >> Thanks
> > >> Raakhi,
> > >>
> > >
> >
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered in
> > Scotland, with registration number SC005336.
> >
>
Re: No space left on device Exception
Posted by Rakhi Khatwani <ra...@gmail.com>.
Thanks,
I will check tht
Regards,
Raakhi
On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne <mi...@inf.ed.ac.uk> wrote:
> it may be that intermediate results are filling your disks and when
> the jobs crash, this all gets deleted. so it would look like you have
> spare space when in reality you don't.
>
> i would check on the file system as your jobs run and see if indeed
> they are filling-up.
>
> Miles
>
> 2009/4/16 Rakhi Khatwani <ra...@gmail.com>:
> > Hi,
> > following is the output on the df command
> > [root@domU-12-31-39-00-E5-D2 conf]# df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/sda1 9.9G 4.2G 5.2G 45% /
> > /dev/sdb 414G 924M 392G 1% /mnt
> >
> > from the o/p it seems that i have quite an amount of memory available.
> but i
> > still get the exception :(
> >
> > Thanks
> > Raakhi
> >
> > On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.desai@hp.com
> >wrote:
> >
> >> From the exception it appears that there is no space left on machine.
> You
> >> can check using 'df'
> >>
> >> Thanks
> >> Milind
> >>
> >> -----Original Message-----
> >> From: Rakhi Khatwani [mailto:rakhi.khatwani@gmail.com]
> >> Sent: Thursday, April 16, 2009 1:15 PM
> >> To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
> >> Subject: No space left on device Exception
> >>
> >> Hi,
> >> I am running a map-reduce program on 6-Node ec2 cluster. and after a
> >> couple of hours all my tasks gets hanged.
> >>
> >> so i started digging into the logs....
> >>
> >> there were no logs for regionserver
> >> no logs for tasktracker.
> >> However for jobtracker i get the following:
> >>
> >> 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> handler 9 on 50002, call
> >> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
> >> true,
> >> 10745) from 10.254.27.79:44222: error: java.io.IOException:
> >> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >> java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException:
> No
> >> space left on device
> >> at
> >>
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> >> at
> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> >> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> >> at
> >> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
> >> at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> >> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
> >> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
> >> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
> >> at java.io.BufferedWriter.close(BufferedWriter.java:248)
> >> at java.io.PrintWriter.close(PrintWriter.java:295)
> >> at
> >>
> >>
> org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
> >> at
> >>
> org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
> >> at org.apache.hadoop.mapred.JobInProgress.comp
> >>
> >>
> >>
> >> following are the disk information on dfs UI
> >> domU-12-31-39-00-0C-A1<
> >>
> http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.8321.19391.360.2
> >> 94.672353 domU-12-31-39-00-16-F1<
> >>
> http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >1In
> >> Service413.380.4621.24391.670.11
> >> 94.752399 domU-12-31-39-00-45-71<
> >>
> http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >1In
> >> Service413.380.6421.34391.40.16
> >> 94.682303 domU-12-31-39-00-E5-D2<
> >>
> http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.6621.53391.180.16
> >> 94.632319 domU-12-31-39-01-64-12<
> >>
> http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >2In
> >> Service413.380.6421.24391.490.16
> >> 94.712264 domU-12-31-39-01-78-D1<
> >>
> http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.4921.24391.650.12
> >> 94.741952
> >>
> >> I m using hadoop 0.19.0 and hbase 0.19.0
> >>
> >> n googling the error i came arcoss the JIRA issue
> >> http://issues.apache.org/jira/browse/HADOOP-4163
> >>
> >> which says tht its been fixed in this version. :(
> >>
> >> Has anyone else come up with this exception?
> >>
> >> how do we check the maximum capacity for usable dfs and non usable dfs.
> >> Thanks
> >> Raakhi,
> >>
> >
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
Re: No space left on device Exception
Posted by Miles Osborne <mi...@inf.ed.ac.uk>.
it may be that intermediate results are filling your disks and when
the jobs crash, this all gets deleted. so it would look like you have
spare space when in reality you don't.
i would check on the file system as your jobs run and see if indeed
they are filling-up.
Miles
2009/4/16 Rakhi Khatwani <ra...@gmail.com>:
> Hi,
> following is the output on the df command
> [root@domU-12-31-39-00-E5-D2 conf]# df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 9.9G 4.2G 5.2G 45% /
> /dev/sdb 414G 924M 392G 1% /mnt
>
> from the o/p it seems that i have quite an amount of memory available. but i
> still get the exception :(
>
> Thanks
> Raakhi
>
> On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <mi...@hp.com>wrote:
>
>> From the exception it appears that there is no space left on machine. You
>> can check using 'df'
>>
>> Thanks
>> Milind
>>
>> -----Original Message-----
>> From: Rakhi Khatwani [mailto:rakhi.khatwani@gmail.com]
>> Sent: Thursday, April 16, 2009 1:15 PM
>> To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
>> Subject: No space left on device Exception
>>
>> Hi,
>> I am running a map-reduce program on 6-Node ec2 cluster. and after a
>> couple of hours all my tasks gets hanged.
>>
>> so i started digging into the logs....
>>
>> there were no logs for regionserver
>> no logs for tasktracker.
>> However for jobtracker i get the following:
>>
>> 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 9 on 50002, call
>> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
>> true,
>> 10745) from 10.254.27.79:44222: error: java.io.IOException:
>> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>> java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
>> space left on device
>> at
>>
>> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>> at
>>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at
>>
>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
>> at
>>
>> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
>> at
>> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
>> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
>> at
>>
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
>> at java.io.DataOutputStream.write(DataOutputStream.java:90)
>> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
>> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
>> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
>> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
>> at java.io.BufferedWriter.close(BufferedWriter.java:248)
>> at java.io.PrintWriter.close(PrintWriter.java:295)
>> at
>>
>> org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
>> at
>> org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
>> at org.apache.hadoop.mapred.JobInProgress.comp
>>
>>
>>
>> following are the disk information on dfs UI
>> domU-12-31-39-00-0C-A1<
>> http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >0In
>> Service413.380.8321.19391.360.2
>> 94.672353 domU-12-31-39-00-16-F1<
>> http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >1In
>> Service413.380.4621.24391.670.11
>> 94.752399 domU-12-31-39-00-45-71<
>> http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >1In
>> Service413.380.6421.34391.40.16
>> 94.682303 domU-12-31-39-00-E5-D2<
>> http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >0In
>> Service413.380.6621.53391.180.16
>> 94.632319 domU-12-31-39-01-64-12<
>> http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >2In
>> Service413.380.6421.24391.490.16
>> 94.712264 domU-12-31-39-01-78-D1<
>> http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> >0In
>> Service413.380.4921.24391.650.12
>> 94.741952
>>
>> I m using hadoop 0.19.0 and hbase 0.19.0
>>
>> n googling the error i came arcoss the JIRA issue
>> http://issues.apache.org/jira/browse/HADOOP-4163
>>
>> which says tht its been fixed in this version. :(
>>
>> Has anyone else come up with this exception?
>>
>> how do we check the maximum capacity for usable dfs and non usable dfs.
>> Thanks
>> Raakhi,
>>
>
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
Re: No space left on device Exception
Posted by Rakhi Khatwani <ra...@gmail.com>.
Hi,
following is the output on the df command
[root@domU-12-31-39-00-E5-D2 conf]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 4.2G 5.2G 45% /
/dev/sdb 414G 924M 392G 1% /mnt
from the o/p it seems that i have quite an amount of memory available. but i
still get the exception :(
Thanks
Raakhi
On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <mi...@hp.com>wrote:
> From the exception it appears that there is no space left on machine. You
> can check using 'df'
>
> Thanks
> Milind
>
> -----Original Message-----
> From: Rakhi Khatwani [mailto:rakhi.khatwani@gmail.com]
> Sent: Thursday, April 16, 2009 1:15 PM
> To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
> Subject: No space left on device Exception
>
> Hi,
> I am running a map-reduce program on 6-Node ec2 cluster. and after a
> couple of hours all my tasks gets hanged.
>
> so i started digging into the logs....
>
> there were no logs for regionserver
> no logs for tasktracker.
> However for jobtracker i get the following:
>
> 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 9 on 50002, call
> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false,
> true,
> 10745) from 10.254.27.79:44222: error: java.io.IOException:
> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
> java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
> space left on device
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> at
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at
>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> at
>
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> at
> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> at
>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
> at java.io.BufferedWriter.close(BufferedWriter.java:248)
> at java.io.PrintWriter.close(PrintWriter.java:295)
> at
>
> org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
> at
> org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
> at org.apache.hadoop.mapred.JobInProgress.comp
>
>
>
> following are the disk information on dfs UI
> domU-12-31-39-00-0C-A1<
> http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >0In
> Service413.380.8321.19391.360.2
> 94.672353 domU-12-31-39-00-16-F1<
> http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >1In
> Service413.380.4621.24391.670.11
> 94.752399 domU-12-31-39-00-45-71<
> http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >1In
> Service413.380.6421.34391.40.16
> 94.682303 domU-12-31-39-00-E5-D2<
> http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >0In
> Service413.380.6621.53391.180.16
> 94.632319 domU-12-31-39-01-64-12<
> http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >2In
> Service413.380.6421.24391.490.16
> 94.712264 domU-12-31-39-01-78-D1<
> http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >0In
> Service413.380.4921.24391.650.12
> 94.741952
>
> I m using hadoop 0.19.0 and hbase 0.19.0
>
> n googling the error i came arcoss the JIRA issue
> http://issues.apache.org/jira/browse/HADOOP-4163
>
> which says tht its been fixed in this version. :(
>
> Has anyone else come up with this exception?
>
> how do we check the maximum capacity for usable dfs and non usable dfs.
> Thanks
> Raakhi,
>
RE: No space left on device Exception
Posted by "Desai, Milind B" <mi...@hp.com>.
>From the exception it appears that there is no space left on machine. You can check using 'df'
Thanks
Milind
-----Original Message-----
From: Rakhi Khatwani [mailto:rakhi.khatwani@gmail.com]
Sent: Thursday, April 16, 2009 1:15 PM
To: hbase-user@hadoop.apache.org; core-user@hadoop.apache.org
Subject: No space left on device Exception
Hi,
I am running a map-reduce program on 6-Node ec2 cluster. and after a
couple of hours all my tasks gets hanged.
so i started digging into the logs....
there were no logs for regionserver
no logs for tasktracker.
However for jobtracker i get the following:
2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 50002, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@2eed7d11, false, true,
10745) from 10.254.27.79:44222: error: java.io.IOException:
org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No
space left on device
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at
org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
at java.io.BufferedWriter.close(BufferedWriter.java:248)
at java.io.PrintWriter.close(PrintWriter.java:295)
at
org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
at
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
at org.apache.hadoop.mapred.JobInProgress.comp
following are the disk information on dfs UI
domU-12-31-39-00-0C-A1<http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.8321.19391.360.2
94.672353 domU-12-31-39-00-16-F1<http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.4621.24391.670.11
94.752399 domU-12-31-39-00-45-71<http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>1In
Service413.380.6421.34391.40.16
94.682303 domU-12-31-39-00-E5-D2<http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.6621.53391.180.16
94.632319 domU-12-31-39-01-64-12<http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>2In
Service413.380.6421.24391.490.16
94.712264 domU-12-31-39-01-78-D1<http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>0In
Service413.380.4921.24391.650.12
94.741952
I m using hadoop 0.19.0 and hbase 0.19.0
n googling the error i came arcoss the JIRA issue
http://issues.apache.org/jira/browse/HADOOP-4163
which says tht its been fixed in this version. :(
Has anyone else come up with this exception?
how do we check the maximum capacity for usable dfs and non usable dfs.
Thanks
Raakhi,