Map output records/reducer input records mismatch

[Vyacheslav Zholudev <vy...@gmail.com>]

Hi,

I'm having multiple hadoop jobs that use the avro mapred API.
Only in one of the jobs I have a visible mismatch between a number of map output records and reducer input records. 

Does anybody encountered such a behavior? Can anybody think of possible explanations of this phenomenon? 

Any pointers/thoughts are highly appreciated!

Best,
Vyacheslav

Re: Map output records/reducer input records mismatch

[Vyacheslav Zholudev <vy...@gmail.com>]

Hi, Anthony,

I'm looking at the reduce input records and map output records.

No, I don't run a combiner (sorry, I've forgotten to mention)

Vyacheslav

On Aug 16, 2011, at 9:25 PM, Anthony Urso wrote:

> Are you looking at "reduce input groups" or "reduce input records"?
> Are you running a combiner?
> 
> On Tue, Aug 16, 2011 at 11:37 AM, Vyacheslav Zholudev
> <vy...@gmail.com> wrote:
>> Hi,
>> 
>> I'm having multiple hadoop jobs that use the avro mapred API.
>> Only in one of the jobs I have a visible mismatch between a number of map output records and reducer input records.
>> 
>> Does anybody encountered such a behavior? Can anybody think of possible explanations of this phenomenon?
>> 
>> Any pointers/thoughts are highly appreciated!
>> 
>> Best,
>> Vyacheslav

Best,
Vyacheslav

Re: Map output records/reducer input records mismatch

[Anthony Urso <an...@cs.ucla.edu>]

Are you looking at "reduce input groups" or "reduce input records"?
Are you running a combiner?

On Tue, Aug 16, 2011 at 11:37 AM, Vyacheslav Zholudev
<vy...@gmail.com> wrote:
> Hi,
>
> I'm having multiple hadoop jobs that use the avro mapred API.
> Only in one of the jobs I have a visible mismatch between a number of map output records and reducer input records.
>
> Does anybody encountered such a behavior? Can anybody think of possible explanations of this phenomenon?
>
> Any pointers/thoughts are highly appreciated!
>
> Best,
> Vyacheslav