You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Praveen Sripati <pr...@gmail.com> on 2013/10/22 10:47:52 UTC
What is significance of the Oozie MR launcher?
Hi,
I created a simple Oozie work flow with Sqoop, Hive and Pig actions. For
each of there actions, Oozie launches a MR launcher and which in turn
launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR jobs
for 3 actions in the work flow.
Why does Oozie start an MR launcher to start the action and not directly
start the action?
Thanks,
Praveen
Re: What is significance of the Oozie MR launcher?
Posted by Serega Sheypak <se...@gmail.com>.
Thanks Robert for sharing.
2013/10/22 Robert Kanter <rk...@cloudera.com>
> It's also to keep the Oozie server from being bogged down or becoming
> unstable. For example, if you have a bunch of workflows running Pig jobs,
> then you'd have the Oozie server running multiple copies of the Pig client
> (which is a relatively "heavy" program) directly. By moving all of the
> user code and external clients to map tasks in the launcher job, the Oozie
> server remains more light-weight and less prone to errors. It can also
> much more scalable this way because the launcher jobs distribute the the
> job launching/monitoring to other machines in the cluster; otherwise, with
> the Oozie server doing everything, we'd have to limit the number of
> concurrent workflows based on your Oozie server's machine specs (RAM, CPU,
> etc). And finally, from an architectural standpoint, the Oozie server
> itself is stateless; that is, everything is stored in the database and the
> Oozie server can be taken down at any point without losing anything. If we
> were to launch jobs directly from the Oozie server, then we'd now have some
> state (e.g. the Pig client cannot be restarted and resumed).
>
>
> - Robert
>
>
> On Tue, Oct 22, 2013 at 7:40 AM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > The other purpose is to have common launch mehanism for all stuff.
> >
> > My tipical workflow brings up to 50MB of additional jars:
> > jars with 3d party pig libs for pig action,
> > jars to interact with metastore via jdbc inside custom java action.
> > It's good approach to put all workflow stuff into HDFS and then run it
> > using oozie. Easy to install, easy to manage.
> >
> > Our typical analytical job runs for 20 minutes (day-window analysis) up
> to
> > 3 hours (3 weeks window). So nobody cares about +20 seconds spent on jar
> > launcher deployment.
> > Try to solve real-life problems and you'll see real problems :)
> >
> >
> > 2013/10/22 Nam Pham <ph...@gmail.com>
> >
> > > The purpose is to distribute jobs to different machines instead of
> > running
> > > all of them on the master node, IMHO.
> > >
> > > Nam
> > >
> > > > On Oct 22, 2013, at 9:24 PM, Praveen Sripati <
> praveensripati@gmail.com
> > >
> > > wrote:
> > > >
> > > > Thanks Serega.
> > > >
> > > > It might be by design, but I don't see any purpose without someone
> > > telling
> > > > me why. It's more of an overhead. As I mentioned I ran a work flow
> with
> > > > three actions and three more launcher MR jobs ran.
> > > >
> > > > Praveen
> > > >
> > > >
> > > > On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com>wrote:
> > > >
> > > >> It's by design. Action is presented as map-only job with fake input.
> > > Oozie
> > > >> packages jar and sends it to HDFS. Then this jar is launched.
> > > >>
> > > >>
> > > >> 2013/10/22 Praveen Sripati <pr...@gmail.com>
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I created a simple Oozie work flow with Sqoop, Hive and Pig
> actions.
> > > For
> > > >>> each of there actions, Oozie launches a MR launcher and which in
> turn
> > > >>> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR
> > > jobs
> > > >>> for 3 actions in the work flow.
> > > >>>
> > > >>> Why does Oozie start an MR launcher to start the action and not
> > > directly
> > > >>> start the action?
> > > >>> Thanks,
> > > >>> Praveen
> > > >>>
> > > >>
> > >
> >
>
Re: What is significance of the Oozie MR launcher?
Posted by Praveen Sripati <pr...@gmail.com>.
Bob,
Thanks for the response, it makes sense.
I ran a work flow with 3 actions in a single node cluster (with 2
map/reduce slots) and the launchers were keeping the map slots busy and
there wasn't any progress in the work flow. And I had to increase the
number of map slots to get thing moving. Then I started thinking why
another MR job is required just for launching an action.
Praveen
On Tue, Oct 22, 2013 at 5:46 PM, Robert Kanter <rk...@cloudera.com> wrote:
> It's also to keep the Oozie server from being bogged down or becoming
> unstable. For example, if you have a bunch of workflows running Pig jobs,
> then you'd have the Oozie server running multiple copies of the Pig client
> (which is a relatively "heavy" program) directly. By moving all of the
> user code and external clients to map tasks in the launcher job, the Oozie
> server remains more light-weight and less prone to errors. It can also
> much more scalable this way because the launcher jobs distribute the the
> job launching/monitoring to other machines in the cluster; otherwise, with
> the Oozie server doing everything, we'd have to limit the number of
> concurrent workflows based on your Oozie server's machine specs (RAM, CPU,
> etc). And finally, from an architectural standpoint, the Oozie server
> itself is stateless; that is, everything is stored in the database and the
> Oozie server can be taken down at any point without losing anything. If we
> were to launch jobs directly from the Oozie server, then we'd now have some
> state (e.g. the Pig client cannot be restarted and resumed).
>
>
> - Robert
>
>
> On Tue, Oct 22, 2013 at 7:40 AM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > The other purpose is to have common launch mehanism for all stuff.
> >
> > My tipical workflow brings up to 50MB of additional jars:
> > jars with 3d party pig libs for pig action,
> > jars to interact with metastore via jdbc inside custom java action.
> > It's good approach to put all workflow stuff into HDFS and then run it
> > using oozie. Easy to install, easy to manage.
> >
> > Our typical analytical job runs for 20 minutes (day-window analysis) up
> to
> > 3 hours (3 weeks window). So nobody cares about +20 seconds spent on jar
> > launcher deployment.
> > Try to solve real-life problems and you'll see real problems :)
> >
> >
> > 2013/10/22 Nam Pham <ph...@gmail.com>
> >
> > > The purpose is to distribute jobs to different machines instead of
> > running
> > > all of them on the master node, IMHO.
> > >
> > > Nam
> > >
> > > > On Oct 22, 2013, at 9:24 PM, Praveen Sripati <
> praveensripati@gmail.com
> > >
> > > wrote:
> > > >
> > > > Thanks Serega.
> > > >
> > > > It might be by design, but I don't see any purpose without someone
> > > telling
> > > > me why. It's more of an overhead. As I mentioned I ran a work flow
> with
> > > > three actions and three more launcher MR jobs ran.
> > > >
> > > > Praveen
> > > >
> > > >
> > > > On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <
> > > serega.sheypak@gmail.com>wrote:
> > > >
> > > >> It's by design. Action is presented as map-only job with fake input.
> > > Oozie
> > > >> packages jar and sends it to HDFS. Then this jar is launched.
> > > >>
> > > >>
> > > >> 2013/10/22 Praveen Sripati <pr...@gmail.com>
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I created a simple Oozie work flow with Sqoop, Hive and Pig
> actions.
> > > For
> > > >>> each of there actions, Oozie launches a MR launcher and which in
> turn
> > > >>> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR
> > > jobs
> > > >>> for 3 actions in the work flow.
> > > >>>
> > > >>> Why does Oozie start an MR launcher to start the action and not
> > > directly
> > > >>> start the action?
> > > >>> Thanks,
> > > >>> Praveen
> > > >>>
> > > >>
> > >
> >
>
Re: What is significance of the Oozie MR launcher?
Posted by Robert Kanter <rk...@cloudera.com>.
It's also to keep the Oozie server from being bogged down or becoming
unstable. For example, if you have a bunch of workflows running Pig jobs,
then you'd have the Oozie server running multiple copies of the Pig client
(which is a relatively "heavy" program) directly. By moving all of the
user code and external clients to map tasks in the launcher job, the Oozie
server remains more light-weight and less prone to errors. It can also
much more scalable this way because the launcher jobs distribute the the
job launching/monitoring to other machines in the cluster; otherwise, with
the Oozie server doing everything, we'd have to limit the number of
concurrent workflows based on your Oozie server's machine specs (RAM, CPU,
etc). And finally, from an architectural standpoint, the Oozie server
itself is stateless; that is, everything is stored in the database and the
Oozie server can be taken down at any point without losing anything. If we
were to launch jobs directly from the Oozie server, then we'd now have some
state (e.g. the Pig client cannot be restarted and resumed).
- Robert
On Tue, Oct 22, 2013 at 7:40 AM, Serega Sheypak <se...@gmail.com>wrote:
> The other purpose is to have common launch mehanism for all stuff.
>
> My tipical workflow brings up to 50MB of additional jars:
> jars with 3d party pig libs for pig action,
> jars to interact with metastore via jdbc inside custom java action.
> It's good approach to put all workflow stuff into HDFS and then run it
> using oozie. Easy to install, easy to manage.
>
> Our typical analytical job runs for 20 minutes (day-window analysis) up to
> 3 hours (3 weeks window). So nobody cares about +20 seconds spent on jar
> launcher deployment.
> Try to solve real-life problems and you'll see real problems :)
>
>
> 2013/10/22 Nam Pham <ph...@gmail.com>
>
> > The purpose is to distribute jobs to different machines instead of
> running
> > all of them on the master node, IMHO.
> >
> > Nam
> >
> > > On Oct 22, 2013, at 9:24 PM, Praveen Sripati <praveensripati@gmail.com
> >
> > wrote:
> > >
> > > Thanks Serega.
> > >
> > > It might be by design, but I don't see any purpose without someone
> > telling
> > > me why. It's more of an overhead. As I mentioned I ran a work flow with
> > > three actions and three more launcher MR jobs ran.
> > >
> > > Praveen
> > >
> > >
> > > On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <
> > serega.sheypak@gmail.com>wrote:
> > >
> > >> It's by design. Action is presented as map-only job with fake input.
> > Oozie
> > >> packages jar and sends it to HDFS. Then this jar is launched.
> > >>
> > >>
> > >> 2013/10/22 Praveen Sripati <pr...@gmail.com>
> > >>
> > >>> Hi,
> > >>>
> > >>> I created a simple Oozie work flow with Sqoop, Hive and Pig actions.
> > For
> > >>> each of there actions, Oozie launches a MR launcher and which in turn
> > >>> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR
> > jobs
> > >>> for 3 actions in the work flow.
> > >>>
> > >>> Why does Oozie start an MR launcher to start the action and not
> > directly
> > >>> start the action?
> > >>> Thanks,
> > >>> Praveen
> > >>>
> > >>
> >
>
Re: What is significance of the Oozie MR launcher?
Posted by Serega Sheypak <se...@gmail.com>.
The other purpose is to have common launch mehanism for all stuff.
My tipical workflow brings up to 50MB of additional jars:
jars with 3d party pig libs for pig action,
jars to interact with metastore via jdbc inside custom java action.
It's good approach to put all workflow stuff into HDFS and then run it
using oozie. Easy to install, easy to manage.
Our typical analytical job runs for 20 minutes (day-window analysis) up to
3 hours (3 weeks window). So nobody cares about +20 seconds spent on jar
launcher deployment.
Try to solve real-life problems and you'll see real problems :)
2013/10/22 Nam Pham <ph...@gmail.com>
> The purpose is to distribute jobs to different machines instead of running
> all of them on the master node, IMHO.
>
> Nam
>
> > On Oct 22, 2013, at 9:24 PM, Praveen Sripati <pr...@gmail.com>
> wrote:
> >
> > Thanks Serega.
> >
> > It might be by design, but I don't see any purpose without someone
> telling
> > me why. It's more of an overhead. As I mentioned I ran a work flow with
> > three actions and three more launcher MR jobs ran.
> >
> > Praveen
> >
> >
> > On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <
> serega.sheypak@gmail.com>wrote:
> >
> >> It's by design. Action is presented as map-only job with fake input.
> Oozie
> >> packages jar and sends it to HDFS. Then this jar is launched.
> >>
> >>
> >> 2013/10/22 Praveen Sripati <pr...@gmail.com>
> >>
> >>> Hi,
> >>>
> >>> I created a simple Oozie work flow with Sqoop, Hive and Pig actions.
> For
> >>> each of there actions, Oozie launches a MR launcher and which in turn
> >>> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR
> jobs
> >>> for 3 actions in the work flow.
> >>>
> >>> Why does Oozie start an MR launcher to start the action and not
> directly
> >>> start the action?
> >>> Thanks,
> >>> Praveen
> >>>
> >>
>
Re: What is significance of the Oozie MR launcher?
Posted by Nam Pham <ph...@gmail.com>.
The purpose is to distribute jobs to different machines instead of running all of them on the master node, IMHO.
Nam
> On Oct 22, 2013, at 9:24 PM, Praveen Sripati <pr...@gmail.com> wrote:
>
> Thanks Serega.
>
> It might be by design, but I don't see any purpose without someone telling
> me why. It's more of an overhead. As I mentioned I ran a work flow with
> three actions and three more launcher MR jobs ran.
>
> Praveen
>
>
> On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <se...@gmail.com>wrote:
>
>> It's by design. Action is presented as map-only job with fake input. Oozie
>> packages jar and sends it to HDFS. Then this jar is launched.
>>
>>
>> 2013/10/22 Praveen Sripati <pr...@gmail.com>
>>
>>> Hi,
>>>
>>> I created a simple Oozie work flow with Sqoop, Hive and Pig actions. For
>>> each of there actions, Oozie launches a MR launcher and which in turn
>>> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR jobs
>>> for 3 actions in the work flow.
>>>
>>> Why does Oozie start an MR launcher to start the action and not directly
>>> start the action?
>>> Thanks,
>>> Praveen
>>>
>>
Re: What is significance of the Oozie MR launcher?
Posted by Praveen Sripati <pr...@gmail.com>.
Thanks Serega.
It might be by design, but I don't see any purpose without someone telling
me why. It's more of an overhead. As I mentioned I ran a work flow with
three actions and three more launcher MR jobs ran.
Praveen
On Tue, Oct 22, 2013 at 9:56 AM, Serega Sheypak <se...@gmail.com>wrote:
> It's by design. Action is presented as map-only job with fake input. Oozie
> packages jar and sends it to HDFS. Then this jar is launched.
>
>
> 2013/10/22 Praveen Sripati <pr...@gmail.com>
>
> > Hi,
> >
> > I created a simple Oozie work flow with Sqoop, Hive and Pig actions. For
> > each of there actions, Oozie launches a MR launcher and which in turn
> > launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR jobs
> > for 3 actions in the work flow.
> >
> > Why does Oozie start an MR launcher to start the action and not directly
> > start the action?
> > Thanks,
> > Praveen
> >
>
Re: What is significance of the Oozie MR launcher?
Posted by Serega Sheypak <se...@gmail.com>.
It's by design. Action is presented as map-only job with fake input. Oozie
packages jar and sends it to HDFS. Then this jar is launched.
2013/10/22 Praveen Sripati <pr...@gmail.com>
> Hi,
>
> I created a simple Oozie work flow with Sqoop, Hive and Pig actions. For
> each of there actions, Oozie launches a MR launcher and which in turn
> launches the action (Sqoop/Hive/Pig). So, there are a total of 6 MR jobs
> for 3 actions in the work flow.
>
> Why does Oozie start an MR launcher to start the action and not directly
> start the action?
> Thanks,
> Praveen
>