You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by jinwei zhu <zh...@gmail.com> on 2013/01/28 10:21:44 UTC
what is the function of "submitLauncher" in JavaActionExecutor.java
Hi, community,
I find that in the method submitLauncher of JavaActionExecutor.java
there is a call to JobClient , it seems to submit a job, but what is the
purpose of submitting the job?
Best regards!
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by jinwei zhu <zh...@gmail.com>.
Hi,harsh, your conclusion is very helpful, thanks ! and also Alejandro.
2013/1/29 Harsh J <ha...@cloudera.com>
> Hi Jinwei,
>
> Oozie submits jobs by first running a 1-map job. The flow is hence:
> Oozie -> JobClient -> Launcher job (on cluster) -> Launcher mapper (on
> cluster) -> JobClient (from map task, on cluster) -> Actual user's
> code/job (on cluster). The real job is submitted from a job on the
> cluster - not from Oozie.
>
> We do this to prevent running any form of user code to run into the
> Oozie runtime itself. For example, if a user's java action code had
> System.exit or other insecure code, we're exposing Oozie to crash as a
> result of having run that code. Hence we run the user code as a job on
> the cluster itself, wherein even if it fails, segfaults, etc., the
> Oozie server remains unaffected, with just the added cost of 1 extra
> map slot.
>
> The LauncherMapper's job itself has no user elements when launching.
> It uses its own I/F and O/F, creates a proper config object and
> invokes the user's class after that.
>
> On Mon, Jan 28, 2013 at 2:51 PM, jinwei zhu <zh...@gmail.com> wrote:
> > Hi, community,
> > I find that in the method submitLauncher of
> JavaActionExecutor.java
> > there is a call to JobClient , it seems to submit a job, but what is the
> > purpose of submitting the job?
> >
> >
> > Best regards!
>
>
>
> --
> Harsh J
>
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by Harsh J <ha...@cloudera.com>.
Hi Jinwei,
Oozie submits jobs by first running a 1-map job. The flow is hence:
Oozie -> JobClient -> Launcher job (on cluster) -> Launcher mapper (on
cluster) -> JobClient (from map task, on cluster) -> Actual user's
code/job (on cluster). The real job is submitted from a job on the
cluster - not from Oozie.
We do this to prevent running any form of user code to run into the
Oozie runtime itself. For example, if a user's java action code had
System.exit or other insecure code, we're exposing Oozie to crash as a
result of having run that code. Hence we run the user code as a job on
the cluster itself, wherein even if it fails, segfaults, etc., the
Oozie server remains unaffected, with just the added cost of 1 extra
map slot.
The LauncherMapper's job itself has no user elements when launching.
It uses its own I/F and O/F, creates a proper config object and
invokes the user's class after that.
On Mon, Jan 28, 2013 at 2:51 PM, jinwei zhu <zh...@gmail.com> wrote:
> Hi, community,
> I find that in the method submitLauncher of JavaActionExecutor.java
> there is a call to JobClient , it seems to submit a job, but what is the
> purpose of submitting the job?
>
>
> Best regards!
--
Harsh J
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by jinwei zhu <zh...@gmail.com>.
Hi, Alejandro, you mean the launcher job is for custom
inputformat/outptuformat ?
Thx!
2013/1/29 Alejandro Abdelnur <tu...@cloudera.com>
> you need always a launcher job not to run user code on job submission in
> the Oozie server (but it the launcher), this is the case if you are using a
> custom inputformat/outptuformat
>
> thx
>
>
> On Mon, Jan 28, 2013 at 5:01 PM, jinwei zhu <zh...@gmail.com> wrote:
>
> > Hi, Alejandro,
> > 1. if the "1 map task " is produced by LaunchMapper ?
> > 2. I have kowned that mapreduceActionExecutor extends
> javaActionExecutor,
> > so mapreduceActionExecutor will also trigger a launcher job of 1 map
> task
> > , meanwhile the mapreduceMain will also trigger a map and reduce task ,
> so
> > why there are two times trigger ?
> >
> > Best regards!
> >
> >
> > 2013/1/29 Alejandro Abdelnur <tu...@gmail.com>
> >
> > > java actions (and mapreduce pig hive sqoop distcp) all trigger a
> launcher
> > > job of 1 map task where the user action runs. in the case of the java
> > > action, in this laincher job the java main class will be run.
> > > thx
> > >
> > > Alejandro
> > > (phone typing)
> > >
> > > On Jan 28, 2013, at 1:21 AM, jinwei zhu <zh...@gmail.com> wrote:
> > >
> > > > Hi, community,
> > > > I find that in the method submitLauncher of
> > JavaActionExecutor.java
> > > > there is a call to JobClient , it seems to submit a job, but what is
> > the
> > > > purpose of submitting the job?
> > > >
> > > >
> > > > Best regards!
> > >
> >
>
>
>
> --
> Alejandro
>
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
you need always a launcher job not to run user code on job submission in
the Oozie server (but it the launcher), this is the case if you are using a
custom inputformat/outptuformat
thx
On Mon, Jan 28, 2013 at 5:01 PM, jinwei zhu <zh...@gmail.com> wrote:
> Hi, Alejandro,
> 1. if the "1 map task " is produced by LaunchMapper ?
> 2. I have kowned that mapreduceActionExecutor extends javaActionExecutor,
> so mapreduceActionExecutor will also trigger a launcher job of 1 map task
> , meanwhile the mapreduceMain will also trigger a map and reduce task , so
> why there are two times trigger ?
>
> Best regards!
>
>
> 2013/1/29 Alejandro Abdelnur <tu...@gmail.com>
>
> > java actions (and mapreduce pig hive sqoop distcp) all trigger a launcher
> > job of 1 map task where the user action runs. in the case of the java
> > action, in this laincher job the java main class will be run.
> > thx
> >
> > Alejandro
> > (phone typing)
> >
> > On Jan 28, 2013, at 1:21 AM, jinwei zhu <zh...@gmail.com> wrote:
> >
> > > Hi, community,
> > > I find that in the method submitLauncher of
> JavaActionExecutor.java
> > > there is a call to JobClient , it seems to submit a job, but what is
> the
> > > purpose of submitting the job?
> > >
> > >
> > > Best regards!
> >
>
--
Alejandro
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by jinwei zhu <zh...@gmail.com>.
Hi, Alejandro,
1. if the "1 map task " is produced by LaunchMapper ?
2. I have kowned that mapreduceActionExecutor extends javaActionExecutor,
so mapreduceActionExecutor will also trigger a launcher job of 1 map task
, meanwhile the mapreduceMain will also trigger a map and reduce task , so
why there are two times trigger ?
Best regards!
2013/1/29 Alejandro Abdelnur <tu...@gmail.com>
> java actions (and mapreduce pig hive sqoop distcp) all trigger a launcher
> job of 1 map task where the user action runs. in the case of the java
> action, in this laincher job the java main class will be run.
> thx
>
> Alejandro
> (phone typing)
>
> On Jan 28, 2013, at 1:21 AM, jinwei zhu <zh...@gmail.com> wrote:
>
> > Hi, community,
> > I find that in the method submitLauncher of JavaActionExecutor.java
> > there is a call to JobClient , it seems to submit a job, but what is the
> > purpose of submitting the job?
> >
> >
> > Best regards!
>
Re: what is the function of "submitLauncher" in JavaActionExecutor.java
Posted by Alejandro Abdelnur <tu...@gmail.com>.
java actions (and mapreduce pig hive sqoop distcp) all trigger a launcher job of 1 map task where the user action runs. in the case of the java action, in this laincher job the java main class will be run.
thx
Alejandro
(phone typing)
On Jan 28, 2013, at 1:21 AM, jinwei zhu <zh...@gmail.com> wrote:
> Hi, community,
> I find that in the method submitLauncher of JavaActionExecutor.java
> there is a call to JobClient , it seems to submit a job, but what is the
> purpose of submitting the job?
>
>
> Best regards!