RCFile and Hadoop Counters

[Avrilia Floratou <fl...@cs.wisc.edu>]

Hi,

I have a question related to the hadoop counters when RCFile is used.
I have 16TB of (uncompressed) data stored in compressed RCFile format. The size of the compressed RCFile is approximately 3 TB.
I ran a simple scan query on this table. Each split is 256 MB (HDFS block size). 

From the counters of each individual map task I can see the following info:

HDFS_BYTES_READ : 91,235,561
Map input bytes: 268,191,006

Then I looked at the aggregate counters produced by the MR job. I see:

HDFS_BYTES_READ :  1,049,781,904,232
Map input bytes:  3,088,881,678,946

The total job time is 4980 sec. During the job I was running iostat to check the bw I was getting from my disks and that was 40 MB/sec at each of my 16
nodes. That means a total of 40*16 = 640 MB/sec across the cluster.

If the raw data read was 1,049,781,904,232 according to the HDFS_BYTES_READ counter then the job would finish in 1640 sec (1TB/ 640mb/sec).
What is wrong here?

I'm actually wondering what these two counters HDFS_BYTES_READ and Map Input Bytes actually represent when compressed RCFiles are used 
as a storage layer and how these are related to the raw bandwidth I can get from iostat.

Thanks,
Avrilia