You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Rahul <rs...@xebia.com> on 2012/06/27 10:12:11 UTC

Pipeline in synchronous mode

Hi Guys,

How can I run Pipeline in synchronous mode ?  I have built junit tests 
but they all run in async mode over MRPipeline.
I need to make sure that the pipeline gets finished so I have to 
continually monitor the output folder.  Is there a better way for this ?

regards,
Rahul

Re: Pipeline in synchronous mode

Posted by Rahul <rs...@xebia.com>.
Hi Gabried,

Thanks this helps. Actually I was getting some exceptions but was unable 
to debug them as the tests got finished outputting the stacktraces.
Later realized some thing silly on my end.

regards
Rahul


On 27-06-2012 15:10, Gabriel Reid wrote:
> On Wed, Jun 27, 2012 at 10:12 AM, Rahul<rs...@xebia.com>  wrote:
>> How can I run Pipeline in synchronous mode ?  I have built junit tests but
>> they all run in async mode over MRPipeline.
>> I need to make sure that the pipeline gets finished so I have to continually
>> monitor the output folder.  Is there a better way for this ?
> Hi Rahul,
>
> Pipelines themselves are always run synchronously; that is, a call to
> Pipeline.run or Pipeline.done will block until all underlying jobs have
> run (or failed).
>
> There shouldn't be a need to monitor output folders; you can count
> on output directories being populated once Pipeline.run or
> Pipeline.done returns (and shouldn't count on the folders being populated
> until that point.
>
> Of course, the underlying MapReduce job(s) are run asynchronously.
> The coordination of the asynchronous MR jobs and the blocking
> of the MRPipeline class is handled by the CrunchJobControl class.
>
> Hope this helps.
>
> Regards,
>
> Gabriel

Re: Pipeline in synchronous mode

Posted by Gabriel Reid <ga...@gmail.com>.
On Wed, Jun 27, 2012 at 10:12 AM, Rahul <rs...@xebia.com> wrote:
> How can I run Pipeline in synchronous mode ?  I have built junit tests but
> they all run in async mode over MRPipeline.
> I need to make sure that the pipeline gets finished so I have to continually
> monitor the output folder.  Is there a better way for this ?

Hi Rahul,

Pipelines themselves are always run synchronously; that is, a call to
Pipeline.run or Pipeline.done will block until all underlying jobs have
run (or failed).

There shouldn't be a need to monitor output folders; you can count
on output directories being populated once Pipeline.run or
Pipeline.done returns (and shouldn't count on the folders being populated
until that point.

Of course, the underlying MapReduce job(s) are run asynchronously.
The coordination of the asynchronous MR jobs and the blocking
of the MRPipeline class is handled by the CrunchJobControl class.

Hope this helps.

Regards,

Gabriel