You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Praveen Sripati <pr...@gmail.com> on 2011/10/01 05:39:17 UTC
How to stop a MR Job when a certain condition is met?
Hi,
Is there a way to stop an entire job when a certain condition is met in the
map/reduce function? Like looking for a particular key or value.
Thanks,
Praveen
Re: How to stop a MR Job when a certain condition is met?
Posted by Praveen Sripati <pr...@gmail.com>.
Arun,
Let's say there is 1PB of data in HDFS and the requirement is to look for a
particular sequence in the whole data. If the sequence is found then the MR
job could be aborted. There is no meaning in keeping the Map/Reduce tasks
running.
There was a query in one of the forums on how a MR job can be aborted if a
particular condition is met.
Thanks,
Praveen
On Sat, Oct 1, 2011 at 9:17 AM, Arun Murthy <ac...@hortonworks.com> wrote:
> Not sure what you are trying, or if MR is the right tool...
>
> Anyway, the basic idea is to write a map-only job with a custom
> RecordReader to stop emitting inputs to your map when the
> mapper/recordreader finds the needle in the haystack.
>
> Arun
>
> Sent from my iPhone
>
> On Sep 30, 2011, at 8:39 PM, Praveen Sripati <pr...@gmail.com>
> wrote:
>
> Hi,
>
> Is there a way to stop an entire job when a certain condition is met in the
> map/reduce function? Like looking for a particular key or value.
>
> Thanks,
> Praveen
>
>
Re: How to stop a MR Job when a certain condition is met?
Posted by Arun Murthy <ac...@hortonworks.com>.
Not sure what you are trying, or if MR is the right tool...
Anyway, the basic idea is to write a map-only job with a custom RecordReader
to stop emitting inputs to your map when the mapper/recordreader finds the
needle in the haystack.
Arun
Sent from my iPhone
On Sep 30, 2011, at 8:39 PM, Praveen Sripati <pr...@gmail.com>
wrote:
Hi,
Is there a way to stop an entire job when a certain condition is met in the
map/reduce function? Like looking for a particular key or value.
Thanks,
Praveen