You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ChaoChun Liang <cc...@gmail.com> on 2007/08/21 09:34:22 UTC
How to run Map/Reduce for iterative algorithms in Hadoop?
Hello:
According the Map/Reduce working flow, it runs map first, then reduce.
Could I run this pairs for several iterations(i.e. to run map/reduce several
times
and stop for certain criteria, like some iterative methods in
numerical-analysis)?
If the answer is yes, any example to do that?
---ChaoChun
--
View this message in context: http://www.nabble.com/How-to-run-Map-Reduce-for-iterative-algorithms-in-Hadoop--tf4303510.html#a12249705
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: How to run Map/Reduce for iterative algorithms in Hadoop?
Posted by Andrzej Bialecki <ab...@getopt.org>.
ChaoChun Liang wrote:
>
> Enis Soztutar wrote:
>> for(int i=0;i<something;i++) {
>> if(isDone())
>> break;
>>
>> JobConf job = new JobConf(conf);
>> job.addInputPath("foo");
>> job.setOutputPath("bar");
>> //set job parameters accordingly
>> JobClient.runJob(job);
>> }
>>
>
> Can I get the result directly from the reduce and decide whether the next
> loop run or not?
> For example, I would like to get the residual value directly from the reduce
> and to run the next loop if this value satisfied some criteria?
You can aggregate some (limited number) values using the Counters
functionality, and retrieve them from the JobTracker when a job completes.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: How to run Map/Reduce for iterative algorithms in Hadoop?
Posted by Ted Dunning <td...@veoh.com>.
There are status global variables that you can set that might help with
that.
On 8/21/07 8:40 AM, "ChaoChun Liang" <cc...@gmail.com> wrote:
>
>
> Enis Soztutar wrote:
>>
>> for(int i=0;i<something;i++) {
>> if(isDone())
>> break;
>>
>> JobConf job = new JobConf(conf);
>> job.addInputPath("foo");
>> job.setOutputPath("bar");
>> //set job parameters accordingly
>> JobClient.runJob(job);
>> }
>>
>
> Can I get the result directly from the reduce and decide whether the next
> loop run or not?
> For example, I would like to get the residual value directly from the reduce
> and to run the next loop if this value satisfied some criteria?
>
> ChaoChun
Re: How to run Map/Reduce for iterative algorithms in Hadoop?
Posted by ChaoChun Liang <cc...@gmail.com>.
Enis Soztutar wrote:
>
> for(int i=0;i<something;i++) {
> if(isDone())
> break;
>
> JobConf job = new JobConf(conf);
> job.addInputPath("foo");
> job.setOutputPath("bar");
> //set job parameters accordingly
> JobClient.runJob(job);
> }
>
Can I get the result directly from the reduce and decide whether the next
loop run or not?
For example, I would like to get the residual value directly from the reduce
and to run the next loop if this value satisfied some criteria?
ChaoChun
--
View this message in context: http://www.nabble.com/How-to-run-Map-Reduce-for-iterative-algorithms-in-Hadoop--tf4303510.html#a12257260
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: How to run Map/Reduce for iterative algorithms in Hadoop?
Posted by Enis Soztutar <en...@gmail.com>.
Yes you can.
If the job you are iterating is the same, then just put the job
submission in a loop. For example :
for(int i=0;i<something;i++) {
if(isDone())
break;
JobConf job = new JobConf(conf);
job.addInputPath("foo");
job.setOutputPath("bar");
//set job parameters accordingly
JobClient.runJob(job);
}
ChaoChun Liang wrote:
> Hello:
>
> According the Map/Reduce working flow, it runs map first, then reduce.
> Could I run this pairs for several iterations(i.e. to run map/reduce several
> times
> and stop for certain criteria, like some iterative methods in
> numerical-analysis)?
> If the answer is yes, any example to do that?
>
> ---ChaoChun
>
>