You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shirley Cohen <sc...@cs.utexas.edu> on 2008/07/29 15:57:45 UTC
iterative map-reduce
Hi,
I want to call a map-reduce program recursively until some condition
is met. How do I do that?
Thanks,
Shirley
Re: iterative map-reduce
Posted by Paco NATHAN <ce...@gmail.com>.
A simple example of Hadoop application code which follows that pattern
(iterate until condition). In the "jyte" section:
http://code.google.com/p/ceteri-mapred/
Loop and condition test are in the same code which calls ToolRunner
and JobClient.
Best,
Paco
On Tue, Jul 29, 2008 at 10:03 AM, Christian Ulrik Søttrup
<so...@nbi.dk> wrote:
> Hi Shirley,
>
> I am basically doing as Qin suggested.
> I am running a job iteratively until some condition is met.
> My main looks something like:(in pseudo code)
>
> main:
> while (!converged):
> make new jobconf
> setup jobconf
> run jobconf
> check reporter for statistics
> decide if converged
>
> I use a custom reporter to check on the fitness of the solution in the
> reduce phase.
>
> If you need more(real java) code drop me a line.
>
> Cheers,
> Christian
Re: iterative map-reduce
Posted by Christian Ulrik Søttrup <so...@nbi.dk>.
Hi Shirley,
I am basically doing as Qin suggested.
I am running a job iteratively until some condition is met.
My main looks something like:(in pseudo code)
main:
while (!converged):
make new jobconf
setup jobconf
run jobconf
check reporter for statistics
decide if converged
I use a custom reporter to check on the fitness of the solution in the
reduce phase.
If you need more(real java) code drop me a line.
Cheers,
Christian
Qin Gao wrote:
> I think it is nothing to do with the framework, just treat the mapredcue as
> a batch process or a subroutine, and you may iteratively call them. If there
> are such interface, I am also interested to know.
>
>
>
> On Tue, Jul 29, 2008 at 10:31 AM, Shirley Cohen <sh...@cis.upenn.edu>wrote:
>
>
>> Thanks... would the iterative script be run outside of Hadoop? I was
>> actually trying to figure out if the framework could handle iterations.
>>
>> Shirley
>>
>>
>> On Jul 29, 2008, at 9:10 AM, Qin Gao wrote:
>>
>> if you are using java, just create job configure again and run it,
>>
>>> otherwise
>>> you just need to write a iterative script.
>>>
>>> On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen <sc...@cs.utexas.edu>
>>> wrote:
>>>
>>> Hi,
>>>
>>>> I want to call a map-reduce program recursively until some condition is
>>>> met. How do I do that?
>>>>
>>>> Thanks,
>>>>
>>>> Shirley
>>>>
>>>>
>>>>
>>>>
>
>
Re: iterative map-reduce
Posted by Qin Gao <qi...@cs.cmu.edu>.
I think it is nothing to do with the framework, just treat the mapredcue as
a batch process or a subroutine, and you may iteratively call them. If there
are such interface, I am also interested to know.
On Tue, Jul 29, 2008 at 10:31 AM, Shirley Cohen <sh...@cis.upenn.edu>wrote:
> Thanks... would the iterative script be run outside of Hadoop? I was
> actually trying to figure out if the framework could handle iterations.
>
> Shirley
>
>
> On Jul 29, 2008, at 9:10 AM, Qin Gao wrote:
>
> if you are using java, just create job configure again and run it,
>> otherwise
>> you just need to write a iterative script.
>>
>> On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen <sc...@cs.utexas.edu>
>> wrote:
>>
>> Hi,
>>>
>>> I want to call a map-reduce program recursively until some condition is
>>> met. How do I do that?
>>>
>>> Thanks,
>>>
>>> Shirley
>>>
>>>
>>>
>
Re: iterative map-reduce
Posted by Shirley Cohen <sh...@cis.upenn.edu>.
Thanks... would the iterative script be run outside of Hadoop? I was
actually trying to figure out if the framework could handle iterations.
Shirley
On Jul 29, 2008, at 9:10 AM, Qin Gao wrote:
> if you are using java, just create job configure again and run it,
> otherwise
> you just need to write a iterative script.
>
> On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen
> <sc...@cs.utexas.edu> wrote:
>
>> Hi,
>>
>> I want to call a map-reduce program recursively until some
>> condition is
>> met. How do I do that?
>>
>> Thanks,
>>
>> Shirley
>>
>>
Re: iterative map-reduce
Posted by Qin Gao <qi...@cs.cmu.edu>.
if you are using java, just create job configure again and run it, otherwise
you just need to write a iterative script.
On Tue, Jul 29, 2008 at 9:57 AM, Shirley Cohen <sc...@cs.utexas.edu> wrote:
> Hi,
>
> I want to call a map-reduce program recursively until some condition is
> met. How do I do that?
>
> Thanks,
>
> Shirley
>
>