You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sandy <sn...@gmail.com> on 2008/09/22 06:33:47 UTC

accessing the number of emitted keys

I have a map that emits <key, value> pairs.

I have a combiner that takes a <key, list(values)> and does something with
it.

Is there a way to get the total number of keys emitted by particular mapper
in the beginning of the combiner function?

Thanks in advance,

-SM

Re: accessing the number of emitted keys

Posted by Sandy <sn...@gmail.com>.
Thanks Owen!

-SM

On Mon, Sep 22, 2008 at 1:02 AM, Owen O'Malley <om...@apache.org> wrote:

>
> On Sep 21, 2008, at 9:33 PM, Sandy wrote:
>
>  Is there a way to get the total number of keys emitted by particular
>> mapper
>> in the beginning of the combiner function?
>>
>
> The short answer is no. As I said in my previous email, the combiner will
> get called when the first spill is being dumped. This can happen while the
> map is still running in a different thread. Therefore, the number wouldn't
> make much sense. Also note that the combiner may be called a second (or
> third or forth) time on a given record as the spills are merged.
>
> -- Owen
>

Re: accessing the number of emitted keys

Posted by Owen O'Malley <om...@apache.org>.
On Sep 21, 2008, at 9:33 PM, Sandy wrote:

> Is there a way to get the total number of keys emitted by particular  
> mapper
> in the beginning of the combiner function?

The short answer is no. As I said in my previous email, the combiner  
will get called when the first spill is being dumped. This can happen  
while the map is still running in a different thread. Therefore, the  
number wouldn't make much sense. Also note that the combiner may be  
called a second (or third or forth) time on a given record as the  
spills are merged.

-- Owen