Versioning of different coordinators for the same oozie job

[Harshal Vora <ha...@komli.com>]

Hi,

Consider a scenario where we submit a coordinator job. After a few
weeks, we want to change the coordinator file or workflow file or
properties file and hence we have to submit a new coordinator job for
the same job. The previous one is either in SUCCESS state or KILLED
state.


Now, consider that after a while we want to re-run instances of this job
for a given time line such that a few instances are from the first
coordinator and others are from the second coordinator.


There might be issues such as frequency of job has changed, code might
not be backward compatible, hadoop cluster configuration has changed
i.e. say we added LZO compression in between etc. Although these issues
are rare, they are worth considering.

Also we will have to rerun dependent jobs.

Has anyone come across such situation? Is there any good solution for
this?

We are brainstorming on ideas for this. I will post these ideas within a
couple of days, but would like to hear from you first and don't want to
bias your ideas towards our ideas.

Regards,

Re: Versioning of different coordinators for the same oozie job

[Harshal Vora <ha...@komli.com>]

Hi Alejandro,

We were thinking on the same lines. 

There are 2 options:
1) To rerun workflow and coordinator instances
For this:
	a. We would never delete any job from oozie database. (for a very long
time).
	b. We need to make sure that our previous coordinators always ended in
SUCCESS state. We cannot kill them, otherwise we cannot rerun (oozie
does not allow).
This will also require us to store which coordinator instance was used
to run job for a particular timestamp.


2) We store some kind of information in our local db's. i.e. the start
time and end time for a coordinator job that was submitted using that
particular version of coordinator.xml, workflow.xml, jars, and the
properties file.

Then when we want to rerun, we spawn a new coordinator using the
versioned files and providing start time and end time for which we want
to rerun.

Both of them are not capable of solving issues that arise due to changes
in the cluster

In my opinion, the second option is much easier.
I will keep you posted on what we plan to do. 


Regards,



On Wed, 2012-07-25 at 14:00 -0700, Alejandro Abdelnur wrote:
> Harshal,
> 
> If you requirement is to be able to rerun jobs using the original app then
> you'll have to keep the COORD and WF dirs in HDFS for all your app
> versions. the simplest way would be to have a 'VERSION' subdir. I.e.:
> 
> /user/tucu/myapps/log-wf/v.001/
> /user/tucu/myapps/log-coord/v.001/
> 
> if you have coordinators and workflows calling subworkflows, you'll have to
> make sure you update the versions in all places. maybe using a job.property
> for that would simplify things.
> 
> Hope this helps. And it would be great, once you nail the problem if you
> would like to write a wiki page explaining the challenges and your solution.
> 
> Thx
> 
> On Mon, Jul 23, 2012 at 3:01 AM, Harshal Vora <ha...@komli.com>wrote:
> 
> > Hi,
> >
> > Consider a scenario where we submit a coordinator job. After a few
> > weeks, we want to change the coordinator file or workflow file or
> > properties file and hence we have to submit a new coordinator job for
> > the same job. The previous one is either in SUCCESS state or KILLED
> > state.
> >
> >
> > Now, consider that after a while we want to re-run instances of this job
> > for a given time line such that a few instances are from the first
> > coordinator and others are from the second coordinator.
> >
> >
> > There might be issues such as frequency of job has changed, code might
> > not be backward compatible, hadoop cluster configuration has changed
> > i.e. say we added LZO compression in between etc. Although these issues
> > are rare, they are worth considering.
> >
> > Also we will have to rerun dependent jobs.
> >
> > Has anyone come across such situation? Is there any good solution for
> > this?
> >
> > We are brainstorming on ideas for this. I will post these ideas within a
> > couple of days, but would like to hear from you first and don't want to
> > bias your ideas towards our ideas.
> >
> > Regards,
> >
> >
> 
> 

Re: Versioning of different coordinators for the same oozie job

[Alejandro Abdelnur <tu...@cloudera.com>]

Harshal,

If you requirement is to be able to rerun jobs using the original app then
you'll have to keep the COORD and WF dirs in HDFS for all your app
versions. the simplest way would be to have a 'VERSION' subdir. I.e.:

/user/tucu/myapps/log-wf/v.001/
/user/tucu/myapps/log-coord/v.001/

if you have coordinators and workflows calling subworkflows, you'll have to
make sure you update the versions in all places. maybe using a job.property
for that would simplify things.

Hope this helps. And it would be great, once you nail the problem if you
would like to write a wiki page explaining the challenges and your solution.

Thx

On Mon, Jul 23, 2012 at 3:01 AM, Harshal Vora <ha...@komli.com>wrote:

> Hi,
>
> Consider a scenario where we submit a coordinator job. After a few
> weeks, we want to change the coordinator file or workflow file or
> properties file and hence we have to submit a new coordinator job for
> the same job. The previous one is either in SUCCESS state or KILLED
> state.
>
>
> Now, consider that after a while we want to re-run instances of this job
> for a given time line such that a few instances are from the first
> coordinator and others are from the second coordinator.
>
>
> There might be issues such as frequency of job has changed, code might
> not be backward compatible, hadoop cluster configuration has changed
> i.e. say we added LZO compression in between etc. Although these issues
> are rare, they are worth considering.
>
> Also we will have to rerun dependent jobs.
>
> Has anyone come across such situation? Is there any good solution for
> this?
>
> We are brainstorming on ideas for this. I will post these ideas within a
> couple of days, but would like to hear from you first and don't want to
> bias your ideas towards our ideas.
>
> Regards,
>
>


-- 
Alejandro