You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sandy <sn...@gmail.com> on 2008/09/22 06:33:47 UTC
accessing the number of emitted keys
I have a map that emits <key, value> pairs.
I have a combiner that takes a <key, list(values)> and does something with
it.
Is there a way to get the total number of keys emitted by particular mapper
in the beginning of the combiner function?
Thanks in advance,
-SM
Re: accessing the number of emitted keys
Posted by Sandy <sn...@gmail.com>.
Thanks Owen!
-SM
On Mon, Sep 22, 2008 at 1:02 AM, Owen O'Malley <om...@apache.org> wrote:
>
> On Sep 21, 2008, at 9:33 PM, Sandy wrote:
>
> Is there a way to get the total number of keys emitted by particular
>> mapper
>> in the beginning of the combiner function?
>>
>
> The short answer is no. As I said in my previous email, the combiner will
> get called when the first spill is being dumped. This can happen while the
> map is still running in a different thread. Therefore, the number wouldn't
> make much sense. Also note that the combiner may be called a second (or
> third or forth) time on a given record as the spills are merged.
>
> -- Owen
>
Re: accessing the number of emitted keys
Posted by Owen O'Malley <om...@apache.org>.
On Sep 21, 2008, at 9:33 PM, Sandy wrote:
> Is there a way to get the total number of keys emitted by particular
> mapper
> in the beginning of the combiner function?
The short answer is no. As I said in my previous email, the combiner
will get called when the first spill is being dumped. This can happen
while the map is still running in a different thread. Therefore, the
number wouldn't make much sense. Also note that the combiner may be
called a second (or third or forth) time on a given record as the
spills are merged.
-- Owen