Last two maps: what happens at the end?

[Fuad Efendi <fu...@efendi.ca>]

Hello,


I am running specific Map/Reduce task on Hadoop cluster 0.19.2, the job was
split on 509 maps, 507 maps run quickly enough, 1-2 minutes each; cluster
capacity: 9 maps, 3 reduces.

The problem is with last 2 maps. I can't monitor it, but it is doing
something at the end. I don't think task is hanged, but I was forced to
increase timeout to even 10 hours; default is 600000 milliseconds. It runs
already 2 hours, I am still waiting.

I can see this in data node logs, and I believe last two maps do something
(sorting?)... "top" command shows constantly 100% for the process (3-core
CPU); some I/O wait time, etc.

2009-12-07 17:55:26,257 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-5446742124105873401_5013
2009-12-07 17:58:26,976 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_943490007545023289_5013
2009-12-07 18:11:55,926 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_1600535934489014079_5013
2009-12-07 18:16:57,888 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-5513951734130352427_5013
2009-12-07 18:26:24,374 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-8095951827886168570_5013
2009-12-07 18:27:28,976 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_8799624630405158602_5013
2009-12-07 18:38:30,432 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_7949352562734117454_5013
2009-12-07 18:43:47,935 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 394 blocks
got processed in 18 msecs
2009-12-07 18:45:05,563 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-7431375870171222932_5013
2009-12-07 18:56:21,270 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-2572350130286370167_5013


Any idea?


Thanks,

Fuad Efendi
+1 416-993-2060
http://www.tokenizer.ca
Data Mining, Vertical Search