You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Harshal <ha...@komli.com> on 2012/02/02 11:54:06 UTC
Change properties at runtime
Hey,
Whenever I submit a new co-ordinator job, I have to provide a properties
file. This properties file also contains the number of reducers that
should run for m-r job of a particular work flow instance.
We might want to change the number of reducers at runtime.
The only option I see right now is to kill the job and submit it again
with a new properties file and adjust the start time such that it will
start processing from the next work flow instance that was supposed to
be processed.
There is another disadvantage to this. If I want to re-run a few jobs
for previous week or previous month, that co-ordinator is already in
KILLED state.
Is there any way to change the properties at runtime?
Regards,
Re: Change properties at runtime
Posted by Harshal <ha...@komli.com>.
In Oozie command line documentation I see the following,
-D <property=value> set/override value for given property
Is this used to override any property that was submitted in
the .properties file?
If so, can you provide the syntax for this?
Regards,
On Mon, 2012-02-06 at 15:25 +0530, Harshal wrote:
> Hi,
>
> In the properties file, I have a parameter named
> numOfReducers=2
>
> and in the M-R action in the workflow.xml, I have set the following
> property.
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>${numOfReducers}</value>
> </property>
>
> Co-ordinator does not have any information regarding number of reducers.
> Is there a better way to set the number of reducers?
>
> Regards,
>
> On Thu, 2012-02-02 at 13:47 -0800, Mohammad Islam wrote:
> > Hi Harshal,
> > Currently there is no easy way of changing the properties after you submit the job.
> >
> > However, I'm trying to find a workaround for your use-case.
> > Question:
> > How do the # of reducers are passed to WF and how it is used in WF/action defintion? Please paste the snippets of relevant coordinator.xml and workflow.xml, will definitely help.
> >
> > Regards,
> > Mohammad
> >
> >
> > ________________________________
> > From: Harshal <ha...@komli.com>
> > To: oozie-users <oo...@incubator.apache.org>
> > Sent: Thursday, February 2, 2012 2:54 AM
> > Subject: Change properties at runtime
> >
> > Hey,
> >
> > Whenever I submit a new co-ordinator job, I have to provide a properties
> > file. This properties file also contains the number of reducers that
> > should run for m-r job of a particular work flow instance.
> >
> > We might want to change the number of reducers at runtime.
> > The only option I see right now is to kill the job and submit it again
> > with a new properties file and adjust the start time such that it will
> > start processing from the next work flow instance that was supposed to
> > be processed.
> >
> > There is another disadvantage to this. If I want to re-run a few jobs
> > for previous week or previous month, that co-ordinator is already in
> > KILLED state.
> >
> > Is there any way to change the properties at runtime?
> >
> > Regards,
>
>
Re: Change properties at runtime
Posted by Harshal <ha...@komli.com>.
Hi,
In the properties file, I have a parameter named
numOfReducers=2
and in the M-R action in the workflow.xml, I have set the following
property.
<property>
<name>mapred.reduce.tasks</name>
<value>${numOfReducers}</value>
</property>
Co-ordinator does not have any information regarding number of reducers.
Is there a better way to set the number of reducers?
Regards,
On Thu, 2012-02-02 at 13:47 -0800, Mohammad Islam wrote:
> Hi Harshal,
> Currently there is no easy way of changing the properties after you submit the job.
>
> However, I'm trying to find a workaround for your use-case.
> Question:
> How do the # of reducers are passed to WF and how it is used in WF/action defintion? Please paste the snippets of relevant coordinator.xml and workflow.xml, will definitely help.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Harshal <ha...@komli.com>
> To: oozie-users <oo...@incubator.apache.org>
> Sent: Thursday, February 2, 2012 2:54 AM
> Subject: Change properties at runtime
>
> Hey,
>
> Whenever I submit a new co-ordinator job, I have to provide a properties
> file. This properties file also contains the number of reducers that
> should run for m-r job of a particular work flow instance.
>
> We might want to change the number of reducers at runtime.
> The only option I see right now is to kill the job and submit it again
> with a new properties file and adjust the start time such that it will
> start processing from the next work flow instance that was supposed to
> be processed.
>
> There is another disadvantage to this. If I want to re-run a few jobs
> for previous week or previous month, that co-ordinator is already in
> KILLED state.
>
> Is there any way to change the properties at runtime?
>
> Regards,
Re: Change properties at runtime
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Harshal,
Currently there is no easy way of changing the properties after you submit the job.
However, I'm trying to find a workaround for your use-case.
Question:
How do the # of reducers are passed to WF and how it is used in WF/action defintion? Please paste the snippets of relevant coordinator.xml and workflow.xml, will definitely help.
Regards,
Mohammad
________________________________
From: Harshal <ha...@komli.com>
To: oozie-users <oo...@incubator.apache.org>
Sent: Thursday, February 2, 2012 2:54 AM
Subject: Change properties at runtime
Hey,
Whenever I submit a new co-ordinator job, I have to provide a properties
file. This properties file also contains the number of reducers that
should run for m-r job of a particular work flow instance.
We might want to change the number of reducers at runtime.
The only option I see right now is to kill the job and submit it again
with a new properties file and adjust the start time such that it will
start processing from the next work flow instance that was supposed to
be processed.
There is another disadvantage to this. If I want to re-run a few jobs
for previous week or previous month, that co-ordinator is already in
KILLED state.
Is there any way to change the properties at runtime?
Regards,