Make reducer task exit early

[Aaron Baff <Aa...@telescope.tv>]

Is there a way to make a Reduce task exit early before it has finished reading all of it's data? Basically I'm doing a group by with a sum, and I only want to return the top 1000 records say. So I have local class int variable to keep track of how many have current been written to the output, and as soon as that is exceeded, simply return at the top of the reduce() function.

Is there any way to optimize it even more to tell the Reduce task, "stop reading data, I don't need any more data"?

--Aaron

Re: Make reducer task exit early

[Yaozhen Pan <it...@gmail.com>]

It can be achieved by overwriting Reducer.run() in new mapreduce API.
But I don't know how to achieve it in old API.

On Sat, Jun 4, 2011 at 8:14 AM, Aaron Baff <Aa...@telescope.tv> wrote:

> Is there a way to make a Reduce task exit early before it has finished
> reading all of it's data? Basically I'm doing a group by with a sum, and I
> only want to return the top 1000 records say. So I have local class int
> variable to keep track of how many have current been written to the output,
> and as soon as that is exceeded, simply return at the top of the reduce()
> function.
>
> Is there any way to optimize it even more to tell the Reduce task, "stop
> reading data, I don't need any more data"?
>
> --Aaron
>