You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2006/10/03 07:12:19 UTC
[jira] Created: (HADOOP-573) Checksum error during sorting in
reducer
Checksum error during sorting in reducer
----------------------------------------
Key: HADOOP-573
URL: http://issues.apache.org/jira/browse/HADOOP-573
Project: Hadoop
Issue Type: Bug
Reporter: Runping Qi
Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.readFully(DataInputStream.java:176)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499868 ]
Doug Cutting commented on HADOOP-573:
-------------------------------------
> I tried replacing my memory. The errors continued.
To be clear, did you replace it with ECC memory?
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500462 ]
Doug Cutting commented on HADOOP-573:
-------------------------------------
> Replaced it with non-ECC memory.
Did you see Dennis Kubes' message?
http://mail-archives.apache.org/mod_mbox/lucene-hadoop-dev/200705.mbox/%3c465C3065.9050501@dragonflymc.com%3e
This has come up before. Folks who have ECC memory don't see checksum errors. Perhaps we should add this to an FAQ.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HADOOP-573) Checksum error during sorting
in reducer
Posted by Dennis Kubes <nu...@dragonflymc.com>.
Trust me when I tell you that this is a hardware problem, most likely a
memory problem. If you are not using ECC memory then it is recommended
that ECC memory be used. We had these same problems over a cluster of
50 machines and all problems disappeared when ECC memory was used. We
spent over a month tracking down and fixing these problems including
spending over 2 weeks combing through the hadoop source code convinced
that it was a software bug. We ran hardware tests on hard disk, cpu,
and memory (which all passed), we tried upgrading to the latest version
of hadoop, and we tried turning off checksum validation in the
configuration. Errors persisted. Hadoop can be especially hard on
memory, most often during sorts, and I don't know why but "good" Non-ECC
memory can sometimes fail intermittently during these jobs. The memory
can test out under other load but fail during hadoop jobs.
What you should be seeing is, depending on the number of machines that
you are running, while the errors occur on many machines they happen
more often on some machines. They should also be happening most often
during major sorting jobs, such as mergesegs and merge crawldb (if you
are using Nutch). If you take one of the machines where the errors
occur frequently and change out its memory for ECC memory you will see
your errors decrease by an order of magnitude. Do it for all of your
machines and the problems will disappear completely. If changing out
for ECC on a single machine doesn't decrease your error rate then I
still think it is a hardware problem and would start looking at the hard
disk or motherboard for that machine.
We would have 20-30 failures for checksums during major jobs. Now we
haven't had a single checksum error in 2-3 weeks of processing (and we
have been running a major job continuously for the last 5 days). The
bad news is this. If you don't change out for ECC memory and you do
have a major hardware issue somewhere, you can end up with corruption on
all replications for a given block. This occurred with our system
shortly before changing out all memory to ECC. With that block we lost
approximately 20-30K urls. If hadoop hadn't implemented the continue
job even if a single map task fails functionality (only recently
implemented and thank you for that) over 5M urls would have been lost.
If this had happened in one of our master databases we could have lost
the master. My point is this, hardware problems are hard to track down
and if they get bad enough can cause a complete failure of the cluster.
If you haven't already done so I suggest searching the hadoop users list
for checksum errors. You will find a thread detailing out trials. Hope
this helps.
Dennis Kubes
Julian Neil (JIRA) wrote:
> [ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499677 ]
>
> Julian Neil commented on HADOOP-573:
> ------------------------------------
>
> I've been seeing the same problem, and other checksum problems. I am somewhat sceptical of the suggestion that is a memory hardware issue, but to be thorough I tried replacing my memory. The errors continued. If there is any additional information I can provide to help track the probem down, please let me know.
> Running on a single Windows Server 2003 (with cygwin) as both namenode and datanode.
> Strangely, some large map/reduce jobs never get checksum errors in the maps or reduces, but one particular job always does.
>
> In addition I have been getting many lost map outputs due to checksum errors. The error usually disappears when the task is retried:
>
> Map output lost, rescheduling: getMapOutput(task_0008_m_000007_0,0) failed :
> org.apache.hadoop.fs.ChecksumException: Checksum error: /tmp/hadoop-sshd_server/mapred/local/task_0008_m_000007_0/file.out at 60215808
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:258)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:211)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1674)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
> at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
> at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
> at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
> at org.mortbay.http.HttpServer.service(HttpServer.java:954)
> at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
> at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
> at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
> at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
> at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
> at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
>
> I'm alse getting errors in final output of the previous map/reduce job which is fed in as input to the next job. These errors do not disappear when the map task retries:
>
> org.apache.hadoop.fs.ChecksumException: Checksum error: hdfs://xxx.xxx.xxx:9900/aa/datamining/deviations_part-00002_step-00001/part-00000 at 13781504
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:258)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:211)
> at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:167)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:93)
> at java.io.DataInputStream.readInt(DataInputStream.java:372)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1523)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1436)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1482)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:73)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:157)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
>
>
>> Checksum error during sorting in reducer
>> ----------------------------------------
>>
>> Key: HADOOP-573
>> URL: https://issues.apache.org/jira/browse/HADOOP-573
>> Project: Hadoop
>> Issue Type: Bug
>> Components: mapred
>> Reporter: Runping Qi
>> Assignee: Owen O'Malley
>>
>> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
>> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
>> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
>> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
>> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
>> at java.io.DataInputStream.readFully(DataInputStream.java:176)
>> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
>> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
>> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
>> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
>> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
>> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
>> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
>> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
>> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
>> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
>> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
>
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Julian Neil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12499677 ]
Julian Neil commented on HADOOP-573:
------------------------------------
I've been seeing the same problem, and other checksum problems. I am somewhat sceptical of the suggestion that is a memory hardware issue, but to be thorough I tried replacing my memory. The errors continued. If there is any additional information I can provide to help track the probem down, please let me know.
Running on a single Windows Server 2003 (with cygwin) as both namenode and datanode.
Strangely, some large map/reduce jobs never get checksum errors in the maps or reduces, but one particular job always does.
In addition I have been getting many lost map outputs due to checksum errors. The error usually disappears when the task is retried:
Map output lost, rescheduling: getMapOutput(task_0008_m_000007_0,0) failed :
org.apache.hadoop.fs.ChecksumException: Checksum error: /tmp/hadoop-sshd_server/mapred/local/task_0008_m_000007_0/file.out at 60215808
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:258)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:211)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:167)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1674)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
I'm alse getting errors in final output of the previous map/reduce job which is fed in as input to the next job. These errors do not disappear when the map task retries:
org.apache.hadoop.fs.ChecksumException: Checksum error: hdfs://xxx.xxx.xxx:9900/aa/datamining/deviations_part-00002_step-00001/part-00000 at 13781504
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:258)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:211)
at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:167)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:93)
at java.io.DataInputStream.readInt(DataInputStream.java:372)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1523)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1436)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1482)
at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:73)
at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:157)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1445)
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-573?page=comments#action_12439670 ]
Runping Qi commented on HADOOP-573:
-----------------------------------
Yes, the task was rescheduled and finished successfully. That is the good news.
The syslog does not have any error reports.
I think it is possible that the problem is due to a software bug. Thus, it is better to leave it open for tracking purpose.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: http://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "notthekarl-misc@yahoo.com (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462175 ]
notthekarl-misc@yahoo.com commented on HADOOP-573:
--------------------------------------------------
Sorry, I should have added that I get the error whilst in the fetch stage using Nutch:
2007-01-03 02:29:15,062 WARN dfs.DistributedFileSystem - Moving bad file C:\tmp\hadoop\mapred\local\reduce_dyohnt\map_0.out to C:\tmp\hadoop\mapred\local\reduce_dyohnt\bad_files\map_0.out.61591645
2007-01-03 02:29:15,093 WARN mapred.LocalJobRunner - job_kh2gl7
org.apache.hadoop.fs.ChecksumException: Checksum error: /tmp/hadoop/mapred/local/reduce_dyohnt/map_0.out at 217448448
at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:123)
at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:99)
K.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assigned To: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-573?page=all ]
Doug Cutting updated HADOOP-573:
--------------------------------
Component/s: mapred
Description:
Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.readFully(DataInputStream.java:176)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
was:
Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.DataInputStream.readFully(DataInputStream.java:176)
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: http://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500771 ]
Doug Cutting commented on HADOOP-573:
-------------------------------------
> I am running tests to see if we should invest in a cluster [ ... ]
FYI, Amazon EC2 provides an easy way to experiment with Hadoop.
http://wiki.apache.org/lucene-hadoop/AmazonEC2
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "notthekarl-misc@yahoo.com (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12462172 ]
notthekarl-misc@yahoo.com commented on HADOOP-573:
--------------------------------------------------
Hi,
I'm getting the same problem but don't know enough about Hadoop to
reschedule the reduce task. Could somebody point me to some
documentation or let me know how to do this?
Many thanks,
K.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assigned To: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Julian Neil (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500087 ]
Julian Neil commented on HADOOP-573:
------------------------------------
No. Sorry, should have been clearer. Replaced it with non-ECC memory. You still think this may be the cause? Can you explain why you think this would fix the problem.
Reading the other similar issues on checksum errors, it appears that files (or their checksum files) at various stages of the map/reduce processing are becoming corrupted when written to disk. There are reports of errors in map output, during sorting, and in reduce output.
It smacks of a tricky threading issue. Both because hadoop is fairly complex in its use of threads, and because the bug is intermittent.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Big Jules (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500558 ]
Big Jules commented on HADOOP-573:
----------------------------------
Apologies again.. I didn't see Dennis' reply.. Will try to convince management to replace with ECC memory. I am running tests to see if we should invest in a cluster and move some of our recommendation system algorithms over to hadoop.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: https://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Runping Qi
> Assignee: Owen O'Malley
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-573) Checksum error during sorting in
reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-573?page=comments#action_12439567 ]
Doug Cutting commented on HADOOP-573:
-------------------------------------
Sort is actually the place where most checksum errors have been reported. I believe this is because sorting keeps data in memory longer than other operations, increasing the chance that it will be corrupted there. Does this node have ECC memory? If so, memory errors are unlikely. Sorting also accounts for a large portion of the number of times data is written to disk, so the corruption could have happened there. It would be worth examining the syslog on that node to see if any disk or memory errors are reported.
I assume the reduce was rescheduled and completed? If so, then I will resolve this issue.
> Checksum error during sorting in reducer
> ----------------------------------------
>
> Key: HADOOP-573
> URL: http://issues.apache.org/jira/browse/HADOOP-573
> Project: Hadoop
> Issue Type: Bug
> Reporter: Runping Qi
>
> Many reduce tasks got killed due to checksum error. The strange thing is that the file was generated by the sort function, and was on a local disk. Here is the stack:
> Checksum error: ../task_0011_r_000140_0/all.2.1 at 5342920704
> at org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:134)
> at org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:110)
> at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:170)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
> at java.io.DataInputStream.readFully(DataInputStream.java:176)
> at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:55)
> at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:89)
> at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1061)
> at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1126)
> at org.apache.hadoop.io.SequenceFile$Reader.nextRaw(SequenceFile.java:1354)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeStream.next(SequenceFile.java:1880)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:1938)
> at org.apache.hadoop.io.SequenceFile$Sorter$MergePass.run(SequenceFile.java:1802)
> at org.apache.hadoop.io.SequenceFile$Sorter.mergePass(SequenceFile.java:1749)
> at org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:1494)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:240)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1066)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira