You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Some Body <so...@squareplanet.de> on 2010/07/13 12:57:09 UTC
block errors
Hi All,
I had a MR job that processed 2000 small (<3MB ea.) files and it took 40 minutes on 8 nodes.
Since the files are small it triggerred 2000 tasks. I packed my 2000 files into a single 445MB
sequence file (K,V == Text,Text == <filename>,<file-content>). The new MR job triggers 7 map
tasks (approx 64MB each) but it takes even longer (49 minutes) so I'm trying to figure out why.
I noticed these errors and I'm hoping someone can shed some light on why?
Before I ran the job I ran hadoop fsck / and everything was healthy.
e.g.: no under-replicated, no corrupt blocks etc.
......
2010-07-13 03:24:20,807 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
2010-07-13 03:24:20,807 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 7 files left.
2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 0 files left.
2010-07-13 03:24:20,814 INFO org.apache.hadoop.mapred.ReduceTask: Merging 7 files, 2401573706 bytes from disk
2010-07-13 03:24:20,815 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2010-07-13 03:24:20,818 INFO org.apache.hadoop.mapred.Merger: Merging 7 sorted segments
2010-07-13 03:24:20,827 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 7 segments left of total size: 2401573678 bytes
2010-07-13 03:30:42,329 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.15.46.73:50010
2010-07-13 03:30:42,329 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_4304053493083580280_260714
2010-07-13 03:31:03,846 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.15.46.35:50010
2010-07-13 03:31:03,846 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3680469905814989852_260716
2010-07-13 03:31:08,233 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.15.46.35:50010
2010-07-13 03:31:08,233 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-673505196560500372_260717
2010-07-13 03:31:14,243 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.15.46.73:50010
2010-07-13 03:31:14,243 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-7054031797345836167_260717
......
Re: block errors
Posted by Ted Yu <yu...@gmail.com>.
Check the datanode log on 10.15.46.73 <http://10.15.46.73:50010/>
You should increase dfs.datanode.max.xcievers
On Tue, Jul 13, 2010 at 3:57 AM, Some Body <so...@squareplanet.de> wrote:
> Hi All,
>
> I had a MR job that processed 2000 small (<3MB ea.) files and it took 40
> minutes on 8 nodes.
> Since the files are small it triggerred 2000 tasks. I packed my 2000 files
> into a single 445MB
> sequence file (K,V == Text,Text == <filename>,<file-content>). The new MR
> job triggers 7 map
> tasks (approx 64MB each) but it takes even longer (49 minutes) so I'm
> trying to figure out why.
>
> I noticed these errors and I'm hoping someone can shed some light on why?
>
> Before I ran the job I ran hadoop fsck / and everything was healthy.
> e.g.: no under-replicated, no corrupt blocks etc.
>
> ......
> 2010-07-13 03:24:20,807 INFO org.apache.hadoop.mapred.ReduceTask:
> GetMapEventsThread exiting
> 2010-07-13 03:24:20,807 INFO org.apache.hadoop.mapred.ReduceTask:
> getMapsEventsThread joined.
> 2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask: Closed
> ram manager
> 2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask:
> Interleaved on-disk merge complete: 7 files left.
> 2010-07-13 03:24:20,808 INFO org.apache.hadoop.mapred.ReduceTask: In-memory
> merge complete: 0 files left.
> 2010-07-13 03:24:20,814 INFO org.apache.hadoop.mapred.ReduceTask: Merging 7
> files, 2401573706 bytes from disk
> 2010-07-13 03:24:20,815 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0
> segments, 0 bytes from memory into reduce
> 2010-07-13 03:24:20,818 INFO org.apache.hadoop.mapred.Merger: Merging 7
> sorted segments
> 2010-07-13 03:24:20,827 INFO org.apache.hadoop.mapred.Merger: Down to the
> last merge-pass, with 7 segments left of total size: 2401573678 bytes
> 2010-07-13 03:30:42,329 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.15.46.73:50010
> 2010-07-13 03:30:42,329 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_4304053493083580280_260714
> 2010-07-13 03:31:03,846 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.15.46.35:50010
> 2010-07-13 03:31:03,846 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_3680469905814989852_260716
> 2010-07-13 03:31:08,233 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.15.46.35:50010
> 2010-07-13 03:31:08,233 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-673505196560500372_260717
> 2010-07-13 03:31:14,243 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.15.46.73:50010
> 2010-07-13 03:31:14,243 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-7054031797345836167_260717
> ......
>
>
>