You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Etienne Dumoulin <et...@idiro.com> on 2012/06/20 13:37:24 UTC

Oozie Synchronous Coordinator

Hi Oozie User,

I am looking at the coordinator documentation:
http://incubator.apache.org/oozie/docs/3.1.3/docs/CoordinatorFunctionalSpec.html

I would like to build a complex workflow, from a daily update.

Maybe I miss something, but I did not find what's happened if I do not have
the daily file.

What I am looking for is, let say there is an incident, the workflow does
not get the daily file,
but get two files the day after. Will it throw an error (sending a mail)?
Will it be smart enough to launch two jobs the day after?

Thanks,

Étienne

Re: Oozie Synchronous Coordinator

Posted by Etienne Dumoulin <et...@idiro.com>.
Hi Maxime,

Thank you very much for your answer.
It is all what I needed to know.

Regards,

Étienne


On 21 June 2012 17:43, Maxime Petazzoni <ma...@turn.com> wrote:

> * Etienne Dumoulin <et...@idiro.com> [2012-06-21 11:10:36]:
>
> > From what Mona pointed out: "the action is materializes when input
> > dependencies are satisfied". If I start a coordinator with a starting
> > date one year earlier that the current date, and I have a daily job with
> > a timeout of -1. Does Oozie creates 365 jobs as soon as the input-event
> > is satisfied ? If the input-event and output-event are satisfied, does
> > oozie try to run the job anyway? If yes, is there a solution to generate
> > the opposite behaviour in the workflow?
>
> I don't think it will start workflows for the backlog. It will just
> start materializing on a go-forward basis the next time the defined
> frequency hits.
>
> > A more independent questions. Is it possible to change the coordinator
> > on the fly?  Let say I have a job running at midnight, I want to make it
> > start at 1 am instead without affecting what is running and the stack of
> > job to run. Is that possible or I have to wait that everything is
> > finished for the date YYYYMMDD, and reload a workflow starting at the
> > date YYYYMM{DD+1}.
>
> You can change a few parameters of a coordinator while it is loaded: the
> end date, the concurrency and set a pause time. See
>
> http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/DG_CommandLineTool.html#Changing_endtimeconcurrencypausetime_of_a_Coordinator_Job
>
> In your case you would have to change the end date of the running
> coordinator to make it stop gracefully and start a new one with the new
> frequency I think.
>
> /Max
> --
> { name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
>  company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }
>

Re: Oozie Synchronous Coordinator

Posted by Maxime Petazzoni <ma...@turn.com>.
* Etienne Dumoulin <et...@idiro.com> [2012-06-21 11:10:36]:

> From what Mona pointed out: "the action is materializes when input
> dependencies are satisfied". If I start a coordinator with a starting
> date one year earlier that the current date, and I have a daily job with
> a timeout of -1. Does Oozie creates 365 jobs as soon as the input-event
> is satisfied ? If the input-event and output-event are satisfied, does
> oozie try to run the job anyway? If yes, is there a solution to generate
> the opposite behaviour in the workflow?

I don't think it will start workflows for the backlog. It will just
start materializing on a go-forward basis the next time the defined
frequency hits.

> A more independent questions. Is it possible to change the coordinator
> on the fly?  Let say I have a job running at midnight, I want to make it
> start at 1 am instead without affecting what is running and the stack of
> job to run. Is that possible or I have to wait that everything is
> finished for the date YYYYMMDD, and reload a workflow starting at the
> date YYYYMM{DD+1}.

You can change a few parameters of a coordinator while it is loaded: the
end date, the concurrency and set a pause time. See
http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/DG_CommandLineTool.html#Changing_endtimeconcurrencypausetime_of_a_Coordinator_Job

In your case you would have to change the end date of the running
coordinator to make it stop gracefully and start a new one with the new
frequency I think.

/Max
-- 
{ name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
  company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }

Re: Oozie Synchronous Coordinator

Posted by Etienne Dumoulin <et...@idiro.com>.
Hi Max, Mona,

Thank you very much, it helps a lot.

If I understand well oozie coordinator. It creates a workflow at a given
time and managed workflows not yet ended with the number of concurrent
job and the execution order.

>From what Mona pointed out: "the action is materializes when input
dependencies are satisfied". If I start a coordinator with a starting
date one year earlier that the current date, and I have a daily job with
a timeout of -1. Does Oozie creates 365 jobs as soon as the input-event
is satisfied ? If the input-event and output-event are satisfied, does
oozie try to run the job anyway? If yes, is there a solution to generate
the opposite behaviour in the workflow?

A more independent questions. Is it possible to change the coordinator
on the fly?  Let say I have a job running at midnight, I want to make it
start at 1 am instead without affecting what is running and the stack of
job to run. Is that possible or I have to wait that everything is
finished for the date YYYYMMDD, and reload a workflow starting at the
date YYYYMM{DD+1}.

Thanks again,

Étienne


On 20 June 2012 18:52, Mona Chitnis <ch...@yahoo-inc.com> wrote:

> Thanks for your response Max. As he correctly pointed out, you can
> configure timeout so that your coordinator waits and materializes an
> action finally when input dependencies are satisfied.
>
> --
> Mona Chitnis
> Oozie dev
> Yahoo!
>
>
>
>
>
> On 6/20/12 10:44 AM, "Maxime Petazzoni" <ma...@turn.com> wrote:
>
> >Hi Etienne,
> >
> >* Etienne Dumoulin <et...@idiro.com> [2012-06-20 12:37:24]:
> >
> >> I am looking at the coordinator documentation:
> >>
> >>
> http://incubator.apache.org/oozie/docs/3.1.3/docs/CoordinatorFunctionalSp
> >>ec.html
> >>
> >> I would like to build a complex workflow, from a daily update.
> >>
> >> Maybe I miss something, but I did not find what's happened if I do not
> >>have
> >> the daily file.
> >>
> >> What I am looking for is, let say there is an incident, the workflow
> >>does
> >> not get the daily file,
> >> but get two files the day after. Will it throw an error (sending a
> >>mail)?
> >> Will it be smart enough to launch two jobs the day after?
> >
> >If I remember and understand correctly, you can set a timeout (in the
> >controls section) to get the appropriate behavior. The timeout is the
> >amount of time, in minutes, that Oozie will wait for the input
> >conditions (here, the presence of your data files) to be satisfied.
> >
> >A value of 0 means no timeout, i.e. your coordinator materialization
> >will fail immediately if the file is not present. -1 means wait forever.
> >Any other ℕ+ value is a timeout in minutes.
> >
> >Finally, note that Oozie never sends emails on its own, and is only as
> >smart as you make it be :)
> >
> >/Max
> >--
> >{ name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
> >  company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }
>
>

Re: Oozie Synchronous Coordinator

Posted by Mona Chitnis <ch...@yahoo-inc.com>.
Thanks for your response Max. As he correctly pointed out, you can
configure timeout so that your coordinator waits and materializes an
action finally when input dependencies are satisfied.

--
Mona Chitnis
Oozie dev
Yahoo!





On 6/20/12 10:44 AM, "Maxime Petazzoni" <ma...@turn.com> wrote:

>Hi Etienne,
>
>* Etienne Dumoulin <et...@idiro.com> [2012-06-20 12:37:24]:
>
>> I am looking at the coordinator documentation:
>> 
>>http://incubator.apache.org/oozie/docs/3.1.3/docs/CoordinatorFunctionalSp
>>ec.html
>> 
>> I would like to build a complex workflow, from a daily update.
>> 
>> Maybe I miss something, but I did not find what's happened if I do not
>>have
>> the daily file.
>> 
>> What I am looking for is, let say there is an incident, the workflow
>>does
>> not get the daily file,
>> but get two files the day after. Will it throw an error (sending a
>>mail)?
>> Will it be smart enough to launch two jobs the day after?
>
>If I remember and understand correctly, you can set a timeout (in the
>controls section) to get the appropriate behavior. The timeout is the
>amount of time, in minutes, that Oozie will wait for the input
>conditions (here, the presence of your data files) to be satisfied.
>
>A value of 0 means no timeout, i.e. your coordinator materialization
>will fail immediately if the file is not present. -1 means wait forever.
>Any other ℕ+ value is a timeout in minutes.
>
>Finally, note that Oozie never sends emails on its own, and is only as
>smart as you make it be :)
>
>/Max
>-- 
>{ name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
>  company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }


Re: Oozie Synchronous Coordinator

Posted by Maxime Petazzoni <ma...@turn.com>.
Hi Etienne,

* Etienne Dumoulin <et...@idiro.com> [2012-06-20 12:37:24]:

> I am looking at the coordinator documentation:
> http://incubator.apache.org/oozie/docs/3.1.3/docs/CoordinatorFunctionalSpec.html
> 
> I would like to build a complex workflow, from a daily update.
> 
> Maybe I miss something, but I did not find what's happened if I do not have
> the daily file.
> 
> What I am looking for is, let say there is an incident, the workflow does
> not get the daily file,
> but get two files the day after. Will it throw an error (sending a mail)?
> Will it be smart enough to launch two jobs the day after?

If I remember and understand correctly, you can set a timeout (in the
controls section) to get the appropriate behavior. The timeout is the
amount of time, in minutes, that Oozie will wait for the input
conditions (here, the presence of your data files) to be satisfied.

A value of 0 means no timeout, i.e. your coordinator materialization
will fail immediately if the file is not present. -1 means wait forever.
Any other ℕ+ value is a timeout in minutes.

Finally, note that Oozie never sends emails on its own, and is only as
smart as you make it be :)

/Max
-- 
{ name: 'Maxime Petazzoni', title: 'Sr. Platform Engineer',
  company: { name: 'Turn, Inc', url: 'http://www.turn.com' } }