You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Tarandeep Singh <ta...@gmail.com> on 2008/09/04 21:48:39 UTC

Stop MR jobs after N records have been produced ?

Hi,

Can I stop Map-Reduce jobs after mappers (or reducers) have produced N
records ?

For example, I am interested in finding any 5 rows in the log files that
have a specific keyword. Once I have got 5 lines, there is no need to check
other lines in the log files and thus Mappers and reducers need to be
stopped.

thanks,
Taran

RE: Stop MR jobs after N records have been produced ?

Posted by Arash Rajaiyan <ar...@yahoo.com>.
bah bah
salaam amir jaan
mibinam ke to ham be hadoop alaghe mandi :D

--- On Fri, 9/5/08, Amir Youssefi <am...@yahoo-inc.com> wrote:
From: Amir Youssefi <am...@yahoo-inc.com>
Subject: RE: Stop MR jobs after N records have been produced ?
To: core-user@hadoop.apache.org
Date: Friday, September 5, 2008, 3:48 AM

Also see following jira for more discussions:

http://issues.apache.org/jira/browse/HADOOP-3973 

I will close this as new interface will address the issue.

- Amir 

-----Original Message-----
From: Owen O'Malley [mailto:owen.omalley@gmail.com] On Behalf Of Owen
O'Malley
Sent: Thursday, September 04, 2008 2:13 PM
To: core-user@hadoop.apache.org
Subject: Re: Stop MR jobs after N records have been produced ?


On Sep 4, 2008, at 12:48 PM, Tarandeep Singh wrote:

> Can I stop Map-Reduce jobs after mappers (or reducers) have produced N

> records ?

You could do this pretty easily by implementing a custom MapRunnable.  
There is no equivalent for reduces. The interface proposed in
HADOOP-1230 would support that kind of application. See:

http://svn.apache.org/repos/asf/hadoop/core/trunk/src/mapred/org/apache/
hadoop/mapreduce/

Look at the new Mapper and Reducer interfaces.

-- Owen

RE: Stop MR jobs after N records have been produced ?

Posted by Amir Youssefi <am...@yahoo-inc.com>.
Also see following jira for more discussions:

http://issues.apache.org/jira/browse/HADOOP-3973 

I will close this as new interface will address the issue.

- Amir 

-----Original Message-----
From: Owen O'Malley [mailto:owen.omalley@gmail.com] On Behalf Of Owen
O'Malley
Sent: Thursday, September 04, 2008 2:13 PM
To: core-user@hadoop.apache.org
Subject: Re: Stop MR jobs after N records have been produced ?


On Sep 4, 2008, at 12:48 PM, Tarandeep Singh wrote:

> Can I stop Map-Reduce jobs after mappers (or reducers) have produced N

> records ?

You could do this pretty easily by implementing a custom MapRunnable.  
There is no equivalent for reduces. The interface proposed in
HADOOP-1230 would support that kind of application. See:

http://svn.apache.org/repos/asf/hadoop/core/trunk/src/mapred/org/apache/
hadoop/mapreduce/

Look at the new Mapper and Reducer interfaces.

-- Owen

Re: Stop MR jobs after N records have been produced ?

Posted by Owen O'Malley <om...@apache.org>.
On Sep 4, 2008, at 12:48 PM, Tarandeep Singh wrote:

> Can I stop Map-Reduce jobs after mappers (or reducers) have produced N
> records ?

You could do this pretty easily by implementing a custom MapRunnable.  
There is no equivalent for reduces. The interface proposed in  
HADOOP-1230 would support that kind of application. See:

http://svn.apache.org/repos/asf/hadoop/core/trunk/src/mapred/org/apache/hadoop/mapreduce/

Look at the new Mapper and Reducer interfaces.

-- Owen