You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by anil gupta <an...@gmail.com> on 2014/12/30 01:37:49 UTC
Run an Oozie workflow after every successful run of another Oozie workflow
Hi,
I have two oozie workflows A and B. Output of A is input to B. A runs many
times a day. I would like to run B everytime there is a successful run of
A.
I can do this very easily if i combine A and B into one workflow.
Unfortunately, i cant do that.
I am still new to Oozie, It seems like i can run B after every successful
run of A using Oozie Coordinator. Can anyone point me to an example or
provide me some hints of how to achieve this in oozie?
--
Thanks & Regards,
Anil Gupta
Re: Run an Oozie workflow after every successful run of another Oozie workflow
Posted by anil gupta <an...@gmail.com>.
Thanks for your suggestion, David & Laurent. Yeah, i finally created a flag
file in every output folder of MR job that A is running. B keeps on
checking for that flag file. If its present then it processes that folder.
ZK solution can be truly event driven but it will require some more work.
On Sat, Jan 3, 2015 at 3:01 PM, David Morel <da...@amakuru.net> wrote:
> At the end of your A workflow, use an FS action to set a flag in a
> date/time named directory somewhere in HDFS, and use that in an input
> synchronous dataset in your B coordinator. It will wait on your A workflow
> completion. Or use zookeeper.
>
> David
> Le 3 janv. 2015 15:38, "Laurent H" <la...@gmail.com> a écrit :
>
> > Hi Anil,
> >
> > I don't have any experience with the following solution but maybe, can
> help
> > you :
> >
> >
> https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.6_Sub-workflow_Action
> >
> > Best Regards,
> >
> > --
> > Laurent HATIER - Consultant Big Data & Business Intelligence chez
> CapGemini
> > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
> >
> > 2014-12-30 1:37 GMT+01:00 anil gupta <an...@gmail.com>:
> >
> > > Hi,
> > >
> > > I have two oozie workflows A and B. Output of A is input to B. A runs
> > many
> > > times a day. I would like to run B everytime there is a successful run
> of
> > > A.
> > > I can do this very easily if i combine A and B into one workflow.
> > > Unfortunately, i cant do that.
> > > I am still new to Oozie, It seems like i can run B after every
> successful
> > > run of A using Oozie Coordinator. Can anyone point me to an example or
> > > provide me some hints of how to achieve this in oozie?
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Run an Oozie workflow after every successful run of another Oozie workflow
Posted by David Morel <da...@amakuru.net>.
Le 4 janv. 2015 00:01, "David Morel" <da...@amakuru.net> a écrit :
>
> At the end of your A workflow, use an FS action to set a flag in a
date/time named directory somewhere in HDFS, and use that in an input
synchronous dataset in your B coordinator. It will wait on your A workflow
completion. Or use zookeeper.
>
> David
I forgot to mention : you can have B coordinators have a set timeout, so if
for a given date/time A didn't run then don't wait forever. This is all
based on the assumption your A workflow runs (or doesn't run) at regular
intervals, like every 10 minutes. Or you could use a single file (no
date/time notion) used as a startup flag for workflow B. In that case, as
soon as B starts it deletes the file, and runs. Next instances of B would
just wait (provided concurrency is set to 1) or die based on the timeout
you set. As I write this I find it a bit clunky, and I think a ZK based
solution would be cleaner :-)
>
> Le 3 janv. 2015 15:38, "Laurent H" <la...@gmail.com> a écrit :
>
>> Hi Anil,
>>
>> I don't have any experience with the following solution but maybe, can
help
>> you :
>>
https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.6_Sub-workflow_Action
>>
>> Best Regards,
>>
>> --
>> Laurent HATIER - Consultant Big Data & Business Intelligence chez
CapGemini
>> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
>> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>>
>> 2014-12-30 1:37 GMT+01:00 anil gupta <an...@gmail.com>:
>>
>> > Hi,
>> >
>> > I have two oozie workflows A and B. Output of A is input to B. A runs
many
>> > times a day. I would like to run B everytime there is a successful run
of
>> > A.
>> > I can do this very easily if i combine A and B into one workflow.
>> > Unfortunately, i cant do that.
>> > I am still new to Oozie, It seems like i can run B after every
successful
>> > run of A using Oozie Coordinator. Can anyone point me to an example or
>> > provide me some hints of how to achieve this in oozie?
>> >
>> > --
>> > Thanks & Regards,
>> > Anil Gupta
>> >
Re: Run an Oozie workflow after every successful run of another Oozie workflow
Posted by David Morel <da...@amakuru.net>.
At the end of your A workflow, use an FS action to set a flag in a
date/time named directory somewhere in HDFS, and use that in an input
synchronous dataset in your B coordinator. It will wait on your A workflow
completion. Or use zookeeper.
David
Le 3 janv. 2015 15:38, "Laurent H" <la...@gmail.com> a écrit :
> Hi Anil,
>
> I don't have any experience with the following solution but maybe, can help
> you :
>
> https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.6_Sub-workflow_Action
>
> Best Regards,
>
> --
> Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>
> 2014-12-30 1:37 GMT+01:00 anil gupta <an...@gmail.com>:
>
> > Hi,
> >
> > I have two oozie workflows A and B. Output of A is input to B. A runs
> many
> > times a day. I would like to run B everytime there is a successful run of
> > A.
> > I can do this very easily if i combine A and B into one workflow.
> > Unfortunately, i cant do that.
> > I am still new to Oozie, It seems like i can run B after every successful
> > run of A using Oozie Coordinator. Can anyone point me to an example or
> > provide me some hints of how to achieve this in oozie?
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
Re: Run an Oozie workflow after every successful run of another Oozie workflow
Posted by Laurent H <la...@gmail.com>.
Hi Anil,
I don't have any experience with the following solution but maybe, can help
you :
https://oozie.apache.org/docs/3.2.0-incubating/WorkflowFunctionalSpec.html#a3.2.6_Sub-workflow_Action
Best Regards,
--
Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
<http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
2014-12-30 1:37 GMT+01:00 anil gupta <an...@gmail.com>:
> Hi,
>
> I have two oozie workflows A and B. Output of A is input to B. A runs many
> times a day. I would like to run B everytime there is a successful run of
> A.
> I can do this very easily if i combine A and B into one workflow.
> Unfortunately, i cant do that.
> I am still new to Oozie, It seems like i can run B after every successful
> run of A using Oozie Coordinator. Can anyone point me to an example or
> provide me some hints of how to achieve this in oozie?
>
> --
> Thanks & Regards,
> Anil Gupta
>