Oozie execution performs actions that are not specified in the workflow

[Alex Shacked <al...@myheritage.com>]

Hi,

We are running Oozie 4.0.0 with a Derby database in a Cloudera 5.1.3
cluster.
The workflow has ~ 400 actions. Workflow sample attached.
The workflow has been running successfuly for many days.
At some point I changed the workflow to run only a small subset of the
originally 400 actions.
At this point Oozie execution did not correspond to the workflow any more.
It ran the actions specified
in the small subset but it also ran many other actions from the original
400 set. It did not run the whole
400 set though.

At first I thought that there was some problem with the new script or
configuration. After many
unsuccessful attempts to solve the problem while changing the script or
configuration,
I discovered that the only way to return to correct execution was to
 completely erase the Derby database
and recreate the database again. After that Oozie started to obey the
workflow specification again.
Also  after recreating the database Oozie performance improved.
The issue did not reproduce but we worry it will hit us again.

I was wondering if you ever encountered something similar. Can you please
point us to some direction
to avoid this issue in the future .

Appreciate your help.

Regards,
Alex

Re: Oozie execution performs actions that are not specified in the workflow

[Robert Kanter <rk...@cloudera.com>]

And as for the performance, it's best not to use Derby other than for
development or testing.  It doesn't scale well and we've seen lots of
performance problems with it in general.  I'd recommend you switch over to
one of the other database backends.

- Robert

On Thu, Apr 30, 2015 at 9:30 AM, Purshotam Shah <
purushah@yahoo-inc.com.invalid> wrote:

> Did you modified the running workflow. What I meant is that the workflow
> was running and you changes it in HDFS.
> If that the case, oozie doesn't support that. You need to kill/suspend the
> running workflow and start a new workflow, which will run modified workflow
> from hdfs.
> Let us know if that is not the case, will dig more.
>
> Puru.
>       From: Alex Shacked <al...@myheritage.com>
>  To: user@oozie.apache.org; Alex Shacked <al...@myheritage.com>
>  Sent: Thursday, April 30, 2015 6:40 AM
>  Subject: Oozie execution performs actions that are not specified in the
> workflow
>
> Hi,
> We are running Oozie 4.0.0 with a Derby database in a Cloudera 5.1.3
> cluster. The workflow has ~ 400 actions. Workflow sample attached.The
> workflow has been running successfuly for many days. At some point I
> changed the workflow to run only a small subset of the originally 400
> actions.At this point Oozie execution did not correspond to the workflow
> any more. It ran the actions specifiedin the small subset but it also ran
> many other actions from the original 400 set. It did not run the whole
> 400 set though.
> At first I thought that there was some problem with the new script or
> configuration. After many
> unsuccessful attempts to solve the problem while changing the script or
> configuration,I discovered that the only way to return to correct execution
> was to  completely erase the Derby database
> and recreate the database again. After that Oozie started to obey the
> workflow specification again.
> Also  after recreating the database Oozie performance improved.
> The issue did not reproduce but we worry it will hit us again.
>
> I was wondering if you ever encountered something similar. Can you please
> point us to some direction
> to avoid this issue in the future .
> Appreciate your help.
> Regards,
> Alex
>
>
>

Re: Oozie execution performs actions that are not specified in the workflow

[Purshotam Shah <pu...@yahoo-inc.com.INVALID>]

Did you modified the running workflow. What I meant is that the workflow was running and you changes it in HDFS.
If that the case, oozie doesn't support that. You need to kill/suspend the running workflow and start a new workflow, which will run modified workflow from hdfs.
Let us know if that is not the case, will dig more.

Puru.
      From: Alex Shacked <al...@myheritage.com>
 To: user@oozie.apache.org; Alex Shacked <al...@myheritage.com> 
 Sent: Thursday, April 30, 2015 6:40 AM
 Subject: Oozie execution performs actions that are not specified in the workflow
   
Hi,
We are running Oozie 4.0.0 with a Derby database in a Cloudera 5.1.3 cluster. The workflow has ~ 400 actions. Workflow sample attached.The workflow has been running successfuly for many days. At some point I changed the workflow to run only a small subset of the originally 400 actions.At this point Oozie execution did not correspond to the workflow any more. It ran the actions specifiedin the small subset but it also ran many other actions from the original 400 set. It did not run the whole 
400 set though.
At first I thought that there was some problem with the new script or configuration. After many 
unsuccessful attempts to solve the problem while changing the script or configuration,I discovered that the only way to return to correct execution was to  completely erase the Derby database
and recreate the database again. After that Oozie started to obey the workflow specification again.
Also  after recreating the database Oozie performance improved. 
The issue did not reproduce but we worry it will hit us again.

I was wondering if you ever encountered something similar. Can you please point us to some direction
to avoid this issue in the future .
Appreciate your help.
Regards,
Alex