You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Michael Lok <fu...@gmail.com> on 2012/02/03 09:34:10 UTC

Async job submission via PigServer

Hi folks,

I was wondering if it's possible to submit register and run a script
using PigServer in MAPREDUCE mode in an asynchronous manner; compared
to how a script is executed right now whereby the program is made to
wait till the job is finished.  Although the task is still executed in
the server, I can't programmatically retrieve the job id for future
reference.


Any help is greatly appreciated.


Thanks.

Re: Async job submission via PigServer

Posted by Daniel Dai <da...@hortonworks.com>.
There is no asynchronous API for Pig. However, Pig does have a
notification mechanism (See PigRunner.run), you can create a separate
thread to simulate the asynchronous call.

Daniel

On Fri, Feb 3, 2012 at 12:34 AM, Michael Lok <fu...@gmail.com> wrote:
> Hi folks,
>
> I was wondering if it's possible to submit register and run a script
> using PigServer in MAPREDUCE mode in an asynchronous manner; compared
> to how a script is executed right now whereby the program is made to
> wait till the job is finished.  Although the task is still executed in
> the server, I can't programmatically retrieve the job id for future
> reference.
>
>
> Any help is greatly appreciated.
>
>
> Thanks.

Re: Async job submission via PigServer

Posted by Michael <fu...@gmail.com>.
Hi Alan,

Thanks for the pointers. Will look at oozie for the time being :)



On Feb 11, 2012, at 2:59, Alan Gates <ga...@hortonworks.com> wrote:

> Not quite yet.  We're working on a web services interface to do this for Pig and other Hadoop tools, but it isn't ready for prime time just yet.  You can see the early form at https://issues.apache.org/jira/browse/HCATALOG-182
> 
> Oozie does have some support for this as well, you could check it out at http://incubator.apache.org/oozie/
> 
> Alan.
> 
> On Feb 3, 2012, at 12:34 AM, Michael Lok wrote:
> 
>> Hi folks,
>> 
>> I was wondering if it's possible to submit register and run a script
>> using PigServer in MAPREDUCE mode in an asynchronous manner; compared
>> to how a script is executed right now whereby the program is made to
>> wait till the job is finished.  Although the task is still executed in
>> the server, I can't programmatically retrieve the job id for future
>> reference.
>> 
>> 
>> Any help is greatly appreciated.
>> 
>> 
>> Thanks.
>

Re: Async job submission via PigServer

Posted by Alan Gates <ga...@hortonworks.com>.
Not quite yet.  We're working on a web services interface to do this for Pig and other Hadoop tools, but it isn't ready for prime time just yet.  You can see the early form at https://issues.apache.org/jira/browse/HCATALOG-182

Oozie does have some support for this as well, you could check it out at http://incubator.apache.org/oozie/

Alan.

On Feb 3, 2012, at 12:34 AM, Michael Lok wrote:

> Hi folks,
> 
> I was wondering if it's possible to submit register and run a script
> using PigServer in MAPREDUCE mode in an asynchronous manner; compared
> to how a script is executed right now whereby the program is made to
> wait till the job is finished.  Although the task is still executed in
> the server, I can't programmatically retrieve the job id for future
> reference.
> 
> 
> Any help is greatly appreciated.
> 
> 
> Thanks.