You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Hamid Oliaei <ol...@gmail.com> on 2012/06/26 10:59:55 UTC
Understanding job completion in other nodes
Hi,
I want to run a job on all of nodes and if one job was completed, the node
must wait until the jobs on the other nodes finish.
For that, every node must signal to all nodes and when every node receives
the signal from every one, next job must be run.
How can I handle that in Hadoop?
Is there any solution to understanding job completion in other nodes?
P.S: I am using Hadoop 0.20.2.
Thanks,
Hamid Oliaei
Oliaei@gmail.com
Re: Understanding job completion in other nodes
Posted by Hamid Oliaei <ol...@gmail.com>.
Hi Christoph ,
I didn't consider waitForCompletion. I'll try using that and hope my
workflow didn't need any additional method.
Thanks a lot.
Hamid Oliaei
Oliaei@gmail.com
AW: Understanding job completion in other nodes
Posted by Christoph Schmitz <ch...@1und1.de>.
Hi Hamid,
I'm not sure if I understand your question correctly, but I think this is exactly what the standard workflow in a Hadoop application looks like:
Job job1 = new Job(...);
// setup job, set Mapper and Reducer, etc.
job1.waitForCompletion(...); // at this point, the cluster will run job 1 and wait for its completion
// think about the results from job 1, plan job 2 accordingly
Job job2 = new Job(...);
// setup another job
job2.waitForCompletion(...); // at this point, the cluster will run job 2 and wait for its completion
etc.
So I think "waitForCompletion" is what you were asking about, right?
Regards,
Christoph
-----Ursprüngliche Nachricht-----
Von: Hamid Oliaei [mailto:oliaei@gmail.com]
Gesendet: Dienstag, 26. Juni 2012 11:00
An: mapreduce-user@hadoop.apache.org
Betreff: Understanding job completion in other nodes
Hi,
I want to run a job on all of nodes and if one job was completed, the node must wait until the jobs on the other nodes finish.
For that, every node must signal to all nodes and when every node receives the signal from every one, next job must be run.
How can I handle that in Hadoop?
Is there any solution to understanding job completion in other nodes?
P.S: I am using Hadoop 0.20.2.
Thanks,
Hamid Oliaei
Oliaei@gmail.com