Massive discrepancies in job's bytes written/read

[Bryan Duxbury <br...@rapleaf.com>]

Hey all,

In looking at the stats for a number of our jobs, the amount of data  
that the UI claims we've read from or written to HDFS is vastly  
larger than the amount of data that should be involved in the job.  
For instance, we have a job that combines small files into big files  
that we're operating on around 2TB worth of data. The outputs in HDFS  
(via hadoop dfs -du) matches the expected size, but the jobtracker UI  
claims that we've read and written around 22TB of data!

By all accounts, Hadoop is actually *doing* the right thing - we're  
not observing excess data reading or writing anywhere. However, this  
massive discrepancy makes the job stats essentially worthless for  
understanding IO in our jobs.

Does anyone know why there's such an enormous difference? Have others  
experienced this problem?

-Bryan

Re: Massive discrepancies in job's bytes written/read

[Owen O'Malley <om...@apache.org>]

On Mar 18, 2009, at 9:28 AM, Bryan Duxbury wrote:

> Is there some area of the codebase that deals with aggregating  
> counters that I should be looking at?

Actually, I take it back. If it was in the aggregation code, I think  
we'd see different counters for other attributes. Let me poke around  
some more.

-- Owen

Re: Massive discrepancies in job's bytes written/read

[Bryan Duxbury <br...@rapleaf.com>]

Is there some area of the codebase that deals with aggregating  
counters that I should be looking at?
-Bryan

On Mar 17, 2009, at 10:20 PM, Owen O'Malley wrote:

>
> On Mar 17, 2009, at 7:44 PM, Bryan Duxbury wrote:
>
>> There is no compression in the mix for us, so that's not the culprit.
>>
>> I'd be sort of willing to believe that spilling and sorting play a  
>> role in this, but, wow, over 10x read and write? That seems like a  
>> big problem.
>
> It happened recently to me too. It was off by 6x. The strange thing  
> was that the individual tasks looked right. It was just the  
> aggregate that was wrong.
>
> -- Owen

Re: Massive discrepancies in job's bytes written/read

[Owen O'Malley <om...@apache.org>]

On Mar 17, 2009, at 7:44 PM, Bryan Duxbury wrote:

> There is no compression in the mix for us, so that's not the culprit.
>
> I'd be sort of willing to believe that spilling and sorting play a  
> role in this, but, wow, over 10x read and write? That seems like a  
> big problem.

It happened recently to me too. It was off by 6x. The strange thing  
was that the individual tasks looked right. It was just the aggregate  
that was wrong.

-- Owen

Re: Massive discrepancies in job's bytes written/read

[Bryan Duxbury <br...@rapleaf.com>]

There is no compression in the mix for us, so that's not the culprit.

I'd be sort of willing to believe that spilling and sorting play a  
role in this, but, wow, over 10x read and write? That seems like a  
big problem.

-Bryan

On Mar 17, 2009, at 6:46 PM, Stefan Will wrote:

> Some of the discrepancy could be due to compression of map input/ 
> output
> format. E.g. The mapper output bytes will show the compressed size,  
> while
> the reduce input will show the uncompressed size. Or something  
> along those
> lines. But I'm also questioning the accuracy of the reporting and  
> suspect
> that some if it is due to all the disk activity that happens while
> processing spills in the mapper and the copy/shuffle/sort phase in the
> reducer. It would certainly be nice if all the byte counts were  
> reported in
> a way that they're comparable.
>
> -- Stefan
>
>
>> From: Bryan Duxbury <br...@rapleaf.com>
>> Reply-To: <co...@hadoop.apache.org>
>> Date: Tue, 17 Mar 2009 17:26:52 -0700
>> To: <co...@hadoop.apache.org>
>> Subject: Massive discrepancies in job's bytes written/read
>>
>> Hey all,
>>
>> In looking at the stats for a number of our jobs, the amount of data
>> that the UI claims we've read from or written to HDFS is vastly
>> larger than the amount of data that should be involved in the job.
>> For instance, we have a job that combines small files into big files
>> that we're operating on around 2TB worth of data. The outputs in HDFS
>> (via hadoop dfs -du) matches the expected size, but the jobtracker UI
>> claims that we've read and written around 22TB of data!
>>
>> By all accounts, Hadoop is actually *doing* the right thing - we're
>> not observing excess data reading or writing anywhere. However, this
>> massive discrepancy makes the job stats essentially worthless for
>> understanding IO in our jobs.
>>
>> Does anyone know why there's such an enormous difference? Have others
>> experienced this problem?
>>
>> -Bryan
>
>

Re: Massive discrepancies in job's bytes written/read

[Stefan Will <st...@gmx.net>]

Some of the discrepancy could be due to compression of map input/output
format. E.g. The mapper output bytes will show the compressed size, while
the reduce input will show the uncompressed size. Or something along those
lines. But I'm also questioning the accuracy of the reporting and suspect
that some if it is due to all the disk activity that happens while
processing spills in the mapper and the copy/shuffle/sort phase in the
reducer. It would certainly be nice if all the byte counts were reported in
a way that they're comparable.

-- Stefan


> From: Bryan Duxbury <br...@rapleaf.com>
> Reply-To: <co...@hadoop.apache.org>
> Date: Tue, 17 Mar 2009 17:26:52 -0700
> To: <co...@hadoop.apache.org>
> Subject: Massive discrepancies in job's bytes written/read
> 
> Hey all,
> 
> In looking at the stats for a number of our jobs, the amount of data
> that the UI claims we've read from or written to HDFS is vastly
> larger than the amount of data that should be involved in the job.
> For instance, we have a job that combines small files into big files
> that we're operating on around 2TB worth of data. The outputs in HDFS
> (via hadoop dfs -du) matches the expected size, but the jobtracker UI
> claims that we've read and written around 22TB of data!
> 
> By all accounts, Hadoop is actually *doing* the right thing - we're
> not observing excess data reading or writing anywhere. However, this
> massive discrepancy makes the job stats essentially worthless for
> understanding IO in our jobs.
> 
> Does anyone know why there's such an enormous difference? Have others
> experienced this problem?
> 
> -Bryan