Oozie works with YARN?

[xeonmailinglist <xe...@gmail.com>]

Hi,

Oozie works with YARN?

Re: Oozie works with YARN?

[Serega Sheypak <se...@gmail.com>]

oozie supports mapreduce actions and java actions.
You can use mapreduce-action for your job and mapreduce or java action to
validate the result.
Then you can use decision node to run fail or success (subsequent mapreduce
action) action.
Oozie does it out of the box.

2015-02-19 0:10 GMT+03:00 xeonmailinglist <xe...@gmail.com>:

> It is a pure map-reduce job.
> Can I create an oozie workflow in java with actions like mentioned?
>
>
>
> On 18-02-2015 21:02, Serega Sheypak wrote:
>
>> what is job?
>> Is it pure map-reduce | pig | hive?
>> Oozie is workflow runner and a kind of the cron (later oozie versions).
>> You can split you job in actions (oozie terminology).
>> 1. run smth
>> 2. validate result of smth
>> 3. decision node: if validation ok, then continue, else fail.
>>
>>
>> 2015-02-18 23:56 GMT+03:00 xeonmailinglist <xe...@gmail.com>:
>>
>>  I want to suspend a mapreduce job when the map tasks finish, execute some
>>> code to validated data, and resume the job.
>>> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I
>>> was in using oozie to do this.
>>>
>>> Is this possible in Oozie?
>>>
>>>
>>>
>>> On 18-02-2015 20:50, Serega Sheypak wrote:
>>>
>>>  are trying to do?
>>>> Generally, it works w/o any problems.
>>>>
>>>>
>>>
>

RE: Oozie works with YARN?

[Kondala Palaka <kp...@gurudatasolutions.com>]

Oozie workflow actions are at the granularity of a map-reduce job and nothing finer.  For what you are looking for, you could create two MR jobs -- the first one with map only (identity reducer) and the second one reduce only (identity mapper) -- and then coordinate them with oozie.

The MR framework doesn't allow for pausing between map and reduce for the simple reason that keeping track of the intermediate results (which are on local disk and NOT on hdfs) and the corresponding jobs will be troublesome. 

My $0.02,
KP

-----Original Message-----
From: xeonmailinglist [mailto:xeonmailinglist@gmail.com] 
Sent: Wednesday, February 18, 2015 1:10 PM
To: user@oozie.apache.org
Subject: Re: Oozie works with YARN?

It is a pure map-reduce job.
Can I create an oozie workflow in java with actions like mentioned?


On 18-02-2015 21:02, Serega Sheypak wrote:
> what is job?
> Is it pure map-reduce | pig | hive?
> Oozie is workflow runner and a kind of the cron (later oozie versions).
> You can split you job in actions (oozie terminology).
> 1. run smth
> 2. validate result of smth
> 3. decision node: if validation ok, then continue, else fail.
>
>
> 2015-02-18 23:56 GMT+03:00 xeonmailinglist <xe...@gmail.com>:
>
>> I want to suspend a mapreduce job when the map tasks finish, execute 
>> some code to validated data, and resume the job.
>> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So 
>> I was in using oozie to do this.
>>
>> Is this possible in Oozie?
>>
>>
>>
>> On 18-02-2015 20:50, Serega Sheypak wrote:
>>
>>> are trying to do?
>>> Generally, it works w/o any problems.
>>>
>>

Re: Oozie works with YARN?

[xeonmailinglist <xe...@gmail.com>]

It is a pure map-reduce job.
Can I create an oozie workflow in java with actions like mentioned?


On 18-02-2015 21:02, Serega Sheypak wrote:
> what is job?
> Is it pure map-reduce | pig | hive?
> Oozie is workflow runner and a kind of the cron (later oozie versions).
> You can split you job in actions (oozie terminology).
> 1. run smth
> 2. validate result of smth
> 3. decision node: if validation ok, then continue, else fail.
>
>
> 2015-02-18 23:56 GMT+03:00 xeonmailinglist <xe...@gmail.com>:
>
>> I want to suspend a mapreduce job when the map tasks finish, execute some
>> code to validated data, and resume the job.
>> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I
>> was in using oozie to do this.
>>
>> Is this possible in Oozie?
>>
>>
>>
>> On 18-02-2015 20:50, Serega Sheypak wrote:
>>
>>> are trying to do?
>>> Generally, it works w/o any problems.
>>>
>>

Re: Oozie works with YARN?

[Serega Sheypak <se...@gmail.com>]

what is job?
Is it pure map-reduce | pig | hive?
Oozie is workflow runner and a kind of the cron (later oozie versions).
You can split you job in actions (oozie terminology).
1. run smth
2. validate result of smth
3. decision node: if validation ok, then continue, else fail.


2015-02-18 23:56 GMT+03:00 xeonmailinglist <xe...@gmail.com>:

> I want to suspend a mapreduce job when the map tasks finish, execute some
> code to validated data, and resume the job.
> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I
> was in using oozie to do this.
>
> Is this possible in Oozie?
>
>
>
> On 18-02-2015 20:50, Serega Sheypak wrote:
>
>> are trying to do?
>> Generally, it works w/o any problems.
>>
>
>

Re: Oozie works with YARN?

[Alejandro Abdelnur <tu...@gmail.com>]

doing that is not possible, you cannot suspend MR jobs mid-flight. When you
suspend an Oozie WF, Oozie will run the action (MR in this case) till
completion and won't start then next one.

HTH

On Wed, Feb 18, 2015 at 12:56 PM, xeonmailinglist <xeonmailinglist@gmail.com
> wrote:

> I want to suspend a mapreduce job when the map tasks finish, execute some
> code to validated data, and resume the job.
> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I
> was in using oozie to do this.
>
> Is this possible in Oozie?
>
>
> On 18-02-2015 20:50, Serega Sheypak wrote:
>
>> are trying to do?
>> Generally, it works w/o any problems.
>>
>
>

Re: Oozie works with YARN?

[David Morel <da...@amakuru.net>]

This is an oozie workflow that you're describing. Maps or whatever else
running, possibly in parallel, then a shell action to produce your excell
sheet (could be java too or anything)… then whatever else you want to run
after. You also have decision nodes that you can use in oozie, and forks. I
don't think you'd want to stop anything down the pipeline.

David
Le 19 févr. 2015 12:03, "xeonmailinglist" <xe...@gmail.com> a
écrit :

> I want to check if the map output data is correct when all the map data is
> correct, and use it in another program. Eg, I am running the wordcount
> example. When all the map tasks finish, another program will read this data
> and put it in an excel sheet. Then, the job will resume.
>
> I really must suspend the job when all the map tasks finish, and resume it
> later.
>
> On 19-02-2015 06:46, David Morel wrote:
>
>> I think you're describing what you imagine is a solution instead of
>> describing what problem you're trying to solve. What would happen if you
>> suspend the job, what difference would it make ? If you describe the use
>> case in more detail that would be more useful.
>>
>
>

Re: Oozie works with YARN?

[Serega Sheypak <se...@gmail.com>]

word-count is map-reduce job.
Map reads words and send them as keys to reduce.
Reduce counts occurrence for each word.
It's impossible to suspend map-reduce execution. What are you trying to
verify?
The only way is only way is to split word-count on two map-reduece jobs:
map-only and mr-job.

Map-only could read words and apply combine function.
Then you can verify output.
Then start mr-job to finally aggregate counters for words.

Looks weird :)

2015-02-19 14:02 GMT+03:00 xeonmailinglist <xe...@gmail.com>:

> I want to check if the map output data is correct when all the map data is
> correct, and use it in another program. Eg, I am running the wordcount
> example. When all the map tasks finish, another program will read this data
> and put it in an excel sheet. Then, the job will resume.
>
> I really must suspend the job when all the map tasks finish, and resume it
> later.
>
>
> On 19-02-2015 06:46, David Morel wrote:
>
>> I think you're describing what you imagine is a solution instead of
>> describing what problem you're trying to solve. What would happen if you
>> suspend the job, what difference would it make ? If you describe the use
>> case in more detail that would be more useful.
>>
>
>

Re: Oozie works with YARN?

[xeonmailinglist <xe...@gmail.com>]

I want to check if the map output data is correct when all the map data 
is correct, and use it in another program. Eg, I am running the 
wordcount example. When all the map tasks finish, another program will 
read this data and put it in an excel sheet. Then, the job will resume.

I really must suspend the job when all the map tasks finish, and resume 
it later.

On 19-02-2015 06:46, David Morel wrote:
> I think you're describing what you imagine is a solution instead of
> describing what problem you're trying to solve. What would happen if you
> suspend the job, what difference would it make ? If you describe the use
> case in more detail that would be more useful.

Re: Oozie works with YARN?

[David Morel <da...@amakuru.net>]

Le 18 févr. 2015 21:57, "xeonmailinglist" <xe...@gmail.com> a
écrit :
>
> I want to suspend a mapreduce job when the map tasks finish, execute some
code to validated data, and resume the job.
> AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I
was in using oozie to do this.
>
> Is this possible in Oozie?

I think you're describing what you imagine is a solution instead of
describing what problem you're trying to solve. What would happen if you
suspend the job, what difference would it make ? If you describe the use
case in more detail that would be more useful.

David

Re: Oozie works with YARN?

[xeonmailinglist <xe...@gmail.com>]

I want to suspend a mapreduce job when the map tasks finish, execute 
some code to validated data, and resume the job.
AFAIK, YARN doesn't have a way to suspend and resume running jobs. So I 
was in using oozie to do this.

Is this possible in Oozie?


On 18-02-2015 20:50, Serega Sheypak wrote:
> are trying to do?
> Generally, it works w/o any problems.

Re: Oozie works with YARN?

[Serega Sheypak <se...@gmail.com>]

What are trying to do?
Generally, it works w/o any problems.

2015-02-18 21:54 GMT+03:00 xeonmailinglist <xe...@gmail.com>:

> Hi,
>
> Oozie works with YARN?
>