You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Hari Sreekumar <hs...@clickable.com> on 2010/11/22 12:02:20 UTC
Getting CheckSumException too often
Hi,
What could be the possible reasons for getting too many checksum
exceptions? I am getting these kind of exceptions quite frequently, and the
whole job fails in the end:
org.apache.hadoop.fs.ChecksumException: Checksum error:
/blk_8186355706212889850:of:/tmp/Webevent_07_05_2010.dat at 4075520
at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1158)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1718)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1770)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Thanks,
Hari
Re: Getting CheckSumException too often
Posted by Steve Loughran <st...@apache.org>.
On 22/11/10 11:02, Hari Sreekumar wrote:
> Hi,
>
> What could be the possible reasons for getting too many checksum
> exceptions? I am getting these kind of exceptions quite frequently, and the
> whole job fails in the end:
>
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> /blk_8186355706212889850:of:/tmp/Webevent_07_05_2010.dat at 4075520
> at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
> at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1158)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:1718)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1770)
> at java.io.DataInputStream.read(DataInputStream.java:83)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
> at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
looks like a warning sign of disk failure -are there other disk health
checks you could run?