You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dan Milstein <dm...@hubteam.com> on 2009/05/01 18:35:19 UTC
Sequence of Streaming Jobs
If I've got a sequence of streaming jobs, each of which depends on the
output of the previous one, is there a good way to launch that
sequence? Meaning, I want step B to only start once step A has
finished.
From within Java JobClient code, I can do submitJob/runJob, but is
there any sort of clean way to do this for a sequence of streaming jobs?
Thanks,
-Dan Milstein
Re: Sequence of Streaming Jobs
Posted by Aditya Desai <ad...@gmail.com>.
In python it can be run as a system command by executing the it in the form
of os.system(cmd). However can anybody tell me how to detect failure of
this command in python.
On Sun, May 3, 2009 at 7:08 AM, Billy Pearson <sa...@pearsonwholesale.com>wrote:
> In php I run exec commands with the job commands and it has a variable that
> stores the exit status code.
>
> Billy
>
>
>
>
> "Mayuran Yogarajah" <ma...@casalemedia.com> wrote in message
> news:49FC975A.3030609@casalemedia.com...
>
> Billy Pearson wrote:
>>
>>> I done this with and array of commands for the jobs in a php script
>>> checking
>>> the return of the job to tell if it failed or not.
>>>
>>> Billy
>>>
>>>
>>> I have this same issue.. How do you check if a job failed or not? You
>> mentioned checking
>> the return code? How are you doing that ?
>>
>> thanks
>>
>>> "Dan Milstein" <dm...@hubteam.com> wrote in
>>> message news:58D66A11-B59C-49F8-B72F-7507482C3B67@hubteam.com...
>>>
>>> If I've got a sequence of streaming jobs, each of which depends on the
>>>> output of the previous one, is there a good way to launch that sequence?
>>>> Meaning, I want step B to only start once step A has finished.
>>>>
>>>> From within Java JobClient code, I can do submitJob/runJob, but is there
>>>> any sort of clean way to do this for a sequence of streaming jobs?
>>>>
>>>> Thanks,
>>>> -Dan Milstein
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
--
James Thurber<http://www.brainyquote.com/quotes/authors/j/james_thurber.html>
- "Women are wiser than men because they know less and understand
more."
Re: Sequence of Streaming Jobs
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
In php I run exec commands with the job commands and it has a variable that
stores the exit status code.
Billy
"Mayuran Yogarajah"
<ma...@casalemedia.com> wrote in message
news:49FC975A.3030609@casalemedia.com...
> Billy Pearson wrote:
>> I done this with and array of commands for the jobs in a php script
>> checking
>> the return of the job to tell if it failed or not.
>>
>> Billy
>>
>>
> I have this same issue.. How do you check if a job failed or not? You
> mentioned checking
> the return code? How are you doing that ?
>
> thanks
>> "Dan Milstein" <dm...@hubteam.com> wrote
>> in
>> message
>> news:58D66A11-B59C-49F8-B72F-7507482C3B67@hubteam.com...
>>
>>> If I've got a sequence of streaming jobs, each of which depends on the
>>> output of the previous one, is there a good way to launch that
>>> sequence?
>>> Meaning, I want step B to only start once step A has finished.
>>>
>>> From within Java JobClient code, I can do submitJob/runJob, but is
>>> there
>>> any sort of clean way to do this for a sequence of streaming jobs?
>>>
>>> Thanks,
>>> -Dan Milstein
>>>
>>>
>>
>>
>>
>
>
Re: Sequence of Streaming Jobs
Posted by jason hadoop <ja...@gmail.com>.
if you are using the sh or bash, the variable $? holds the exit status of
the last command to execute.
hadoop jar streaming.jar ...
if [ $? -ne 0 ]; then
echo "My job failed" 2>&1
exit 1
fi
Caution $? is the very last command to execute's exit status. It is easy to
run another command before testing and then test the wrong command's exit
status
On Sat, May 2, 2009 at 11:56 AM, Mayuran Yogarajah <
mayuran.yogarajah@casalemedia.com> wrote:
> Billy Pearson wrote:
>
>> I done this with and array of commands for the jobs in a php script
>> checking
>> the return of the job to tell if it failed or not.
>>
>> Billy
>>
>>
>>
> I have this same issue.. How do you check if a job failed or not? You
> mentioned checking
> the return code? How are you doing that ?
>
> thanks
>
> "Dan Milstein" <dm...@hubteam.com> wrote in
>> message news:58D66A11-B59C-49F8-B72F-7507482C3B67@hubteam.com...
>>
>>
>>> If I've got a sequence of streaming jobs, each of which depends on the
>>> output of the previous one, is there a good way to launch that sequence?
>>> Meaning, I want step B to only start once step A has finished.
>>>
>>> From within Java JobClient code, I can do submitJob/runJob, but is there
>>> any sort of clean way to do this for a sequence of streaming jobs?
>>>
>>> Thanks,
>>> -Dan Milstein
>>>
>>>
>>>
>>
>>
>>
>>
>
>
--
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
Re: Sequence of Streaming Jobs
Posted by Mayuran Yogarajah <ma...@casalemedia.com>.
Billy Pearson wrote:
> I done this with and array of commands for the jobs in a php script checking
> the return of the job to tell if it failed or not.
>
> Billy
>
>
I have this same issue.. How do you check if a job failed or not? You
mentioned checking
the return code? How are you doing that ?
thanks
> "Dan Milstein" <dm...@hubteam.com> wrote in
> message news:58D66A11-B59C-49F8-B72F-7507482C3B67@hubteam.com...
>
>> If I've got a sequence of streaming jobs, each of which depends on the
>> output of the previous one, is there a good way to launch that sequence?
>> Meaning, I want step B to only start once step A has finished.
>>
>> From within Java JobClient code, I can do submitJob/runJob, but is there
>> any sort of clean way to do this for a sequence of streaming jobs?
>>
>> Thanks,
>> -Dan Milstein
>>
>>
>
>
>
Re: Sequence of Streaming Jobs
Posted by Billy Pearson <sa...@pearsonwholesale.com>.
I done this with and array of commands for the jobs in a php script checking
the return of the job to tell if it failed or not.
Billy
"Dan Milstein" <dm...@hubteam.com> wrote in
message news:58D66A11-B59C-49F8-B72F-7507482C3B67@hubteam.com...
> If I've got a sequence of streaming jobs, each of which depends on the
> output of the previous one, is there a good way to launch that sequence?
> Meaning, I want step B to only start once step A has finished.
>
> From within Java JobClient code, I can do submitJob/runJob, but is there
> any sort of clean way to do this for a sequence of streaming jobs?
>
> Thanks,
> -Dan Milstein
>